Benchmarks improve but conditions for replacing junior engineers remain unmet

The latest METR data show that the time horizon—the length of time a model succeeds on 50 % of tasks—has doubled every seven months from 2019 to 2025, and the rate accelerated in 2024-2026. However, 50 % success does not enable staffing a role; the 80 % horizon is dramatically shorter, and models approach near-perfect performance only on tasks lasting less than four human minutes, while on tasks longer than four hours they fail in 90 % of cases. More importantly, METR tasks are designed to be autonomous and well-defined, which a person without prior context can complete in two hours, whereas the first half-year for a junior is devoted entirely to acquiring context: which service is responsible for what, why a particular abstraction exists, and who to contact. The benchmark measures precisely the portion of work from which the real difficulty is stripped.

When the lab that created the test gave up on it, in February 2026 OpenAI stopped reporting SWE-bench Verified and advised others to do the same. In a review of 27.6 % of the dataset it was discovered that at least 59.4 % of the examined problems contained faulty cases that rejected correct solutions, and contamination was found: frontier models reproduced exact patches and problem details word-for-word, evidence of training exposure. The official result rose from 74.9 % to 80.9 % in six months, but the remaining failures were largely due to dataset properties rather than model limitations. In moving to harder, cleaner suites—SWE-bench Pro, Terminal-Bench, and new long-horizon benchmarks—the scores fall far below the numbers highlighted in launch posts.

Cost of verification versus cost of delegation is the third, often neglected, condition: the cost of checking an agent’s output must drop below the cost of delegating the work to a human. METR ran a controlled random experiment with 16 experienced open-source engineers, and the empirical evidence is the cleanest available. When the model generates code that appears correct but contains subtle bugs, the review time of a senior engineer exceeds the time it would have taken to write the code from scratch. As long as this gap exists, the model remains an assistance tool, not a replacement.

The author points to a fourth condition that should cause concern precisely because it does not require the first three. Even if models remain unreliable on long tasks, benchmarks stay biased, and verification stays expensive, organizations may still decide to replace juniors for budgetary or managerial pressure reasons. This is an organizational, not a technological, decision and may be taken long before the technical conditions mature.