AI capability gains accelerate, but only where correct answers are easy to verify

A new analysis by Epoch AI confirms a suspicion that has been circulating in labs for months: the rate of improvement in models has accelerated markedly since the emergence of reasoning models in 2024. The researchers examined four different capability metrics – Epoch’s ECI, METR’s horizon in version 1.1, a composite mathematics metric they built themselves, and results from WeirdML V2 – and observed acceleration in three of them. The fourth metric continued to follow a linear trend.

The team fitted several candidate curves – a simple linear function, a hyperbolic function, and a pair of separate linear segments – to historical data from 2022 onward and evaluated which best predicted unseen data via cross-validation. The winning model for the three accelerated metrics was the same pair of separate linear segments: one for reasoning models and one for all other models. This implies not merely a one-off jump with the new generation, but a rapid improvement rate two to three times faster than the previous trajectory.

The authors note that the split between “reasoning” and “non-reasoning” is not a clean dichotomy. In the same few-month window at the end of 2023, several coordinated changes occurred: a dramatic increase in inference compute, heavier use of reinforcement learning (RL) during post-training, and models that emit explicit reasoning tokens. Isolating which factor drove the acceleration is difficult, but the timing is unequivocal.

WeirdML V2, the fourth metric, fits best to a single linear line with no sign of a break. A possible explanation is that the benchmark places models in a resource-constrained environment – only five attempts to submit working code, with no access to external tools – which differs from the setting used to train recent reasoning models with RL. Moreover, the metric has just about a year of pre-reasoning data, limiting its statistical power to detect a break from the outset.

The three accelerated metrics focus on programming and mathematics, domains that labs have explicitly earmarked for improvement and that share a critical property: easy automatic verification of correctness. This makes them natural targets for RL. Tasks where correctness is harder to verify – open-ended writing, legal analysis, system design – may not have enjoyed the same jump. The headline numbers could be misleading if taken as universal.

The researchers acknowledge that the limiting factor is the availability of benchmarks covering a sufficiently broad range of model versions. They intend to extend the analysis to areas beyond programming and mathematics. Until then, the cautious conclusion is that measurable acceleration is tied to the rise of reasoning models and is concentrated where RL can operate most effectively; the rest of the capability landscape continues at the prior pace.