New paper shows how to evaluate production agents at a fraction of the cost

Researchers reconstructed the full benchmark score for an active analytics agent serving tens of thousands of monthly users with just 38.5% of the questions — a deviation of only 1.03 points. The preprint, posted to arXiv as 2609.21267, has not yet been peer-reviewed. The method was built on 574 historical runs split into calibration and holdout windows by date.

The team compared four approaches: random sampling, cached results, fixed representative subsets, and adaptive testing based on Item Response Theory (IRT). The multi-dimensional adaptive approach under a 2PL model achieved the highest fidelity to the full score. In production, however, the team chose fixed difficulty-stratified subsets because they are simpler to run and maintain.

The surprise: those same fixed subsets transferred to five other agent families without recalibration and remained stable even when the calibration window shrank to a single day. That means a compact evaluation set can be built once and reused over time and across architectures — a meaningful savings in compute and engineering time.

For teams running nightly or weekly evaluations on production agents, the finding offers a practical path: build a difficulty-stratified set of roughly 200 items, calibrate it on a short history, and run only that. The paper is available with an interactive chat at academy.dair.ai/papers/efficie. Review the methodology before adopting; it remains a preprint, not a fully peer-reviewed study.