Agent harness becomes the real test, not the task

HarnessDev inverts the standard evaluation loop. Instead of counting how many tasks an agent solves, it measures how well the model builds the infrastructure that lets it solve them. The harness — the execution layer, tooling, repair loops and memory wrappers around the model — now determines performance as much as the weights themselves. The paper (arXiv 2609.01437) defines two phases: Creation, where the model receives a minimal seed and a few examples and builds a full execution system; and Evolution, where it improves that system using feedback from runs on downstream benchmarks. The metrics are capability (success on held-out tasks) and efficiency (inference token cost).

Six creator models, four domains — code, search and research, writing, ML experiments — and five downstream benchmarks produced 2,207 unique instances with hidden evaluation tasks never exposed during development. Results are uneven. In code and search/research, generated harnesses still trail mature human references by a wide margin. In writing and ML experiments they match or exceed those references. Inference cost varies wildly; some harnesses burn tokens with no proportional gain.

Evolution delivered gains, but they are unstable and do not transfer well to held-out tasks. When researchers fixed the inference model and tested harnesses built by other models, the improvements proved tightly coupled to the specific model running them. A harness developed by Model A does not necessarily help Model B. The conclusion: optimization is bound to the particular model, and cross-architecture generalization is limited.

HarnessDev is not alone. The paper’s bibliography lists HarnessOpt-Bench, Evo-Bench, DarwinX, MemoHarness, StarHarness, OneDayAgent and TTHe, all from 2026. The field is converging on the harness as a critical parameter that deserves separate evaluation, and on test-time harness evolution as a potentially cheaper alternative to retraining. This paper adds a clean split between initial construction and iterative improvement, plus the dual measurement of capability versus cost.

For teams shipping agents in production: do not stop at the final-task benchmark. Measure how much engineering effort your harness demands, how many tokens it burns, and whether it survives a model swap when you change provider or version. HarnessDev offers a first framework for that comparison, but the gaps against human engineering in code and search show that full automation of the infrastructure layer remains a long way off.