New benchmark reveals coding agents fail at building other agents

Researchers have published a benchmark called AgentSWE that tests whether coding agents can build, edit, and optimize other agents. The results are not encouraging: existing systems fall far short.

The benchmark covers 25 tasks across three categories — creation, editing, and optimization. In the creation category, 50 runs produced an average score of 11.2 out of 100, with 20 runs scoring absolute zero. The gap between the average and the maximum score indicates that current agents struggle even to begin tasks that involve building other agents; 11.2 out of 100 is not a partial failure, it is a systemic one.

One commenter proposed separating "validity-gate" failures — cases where the solution never gets off the ground — from low task-quality scores in the reporting. The argument is that a shipment that never runs and a shipment that runs but misses the requirement demand fundamentally different fixes, even though both contribute zero to the overall average.

That distinction speaks to a broader evaluation-design problem: a single average obscures fundamentally different failure modes. Without that breakdown, developers get a misleading picture of where exactly the system gets stuck — whether in basic execution problems or in fulfilling the requirements themselves.