Same model, different harness yields dramatically different results

A new paper posted as a preprint on arXiv shows that what determines how well a coding agent solves tasks is not only the model but also the “harness” that wraps it. The harness is the system that decides what the model sees, which tools it can use, and how the work proceeds. Researchers kept the model and the tasks fixed and swapped only the harness configuration. The result: a two-digit improvement in success metrics without touching the weights at all.

The experiment compared two harness configurations. The control configuration feeds the model the entire conversation in full chronological order. The treatment configuration keeps the same transcript but mechanically truncates results of old tools when the context fills up, and intervenes when the work stalls or repeats. Three benchmarks were used: SWE-bench Verified, SWE-bench Pro and FeatureBench. The main run used a dense context window of 20,480 tokens, a time limit of 480 seconds per attempt, and 169 Verified tasks.

Under those conditions the treatment raised the average per-task failure-to-success transition rate (F2PF) from 28 % to 49 %, and the number of full solutions from 43 to 72. This is not a marginal gain; it shifts performance from “barely works” to “fully solves”. Importantly, the same frozen treatment configuration, without any model-specific retuning, improved both metrics on three additional models with different architectures, indicating that the mechanism generalizes rather than being a point-specific tweak.

When the context window was expanded and Qwen3.6 was evaluated, the Verified and Pro results were close between the two configurations, while FeatureBench retained a higher F2PF under treatment. Another observation: in the larger window the treatment consumed fewer prompt tokens per turn, which makes sense—when history is managed better the model need not be fed all accumulated “junk”.

The paper argues, correctly, that evaluations of coding agents must treat the model + harness pair as a single unit. Publishing a SWE-bench score without stating which harness ran it is like reporting an engine’s fuel consumption without naming the vehicle. The harness is not a negligible technical detail; it is part of the solver.

Caveats: the work is a preprint and has not undergone peer review. Experiments cover three specific benchmarks and a limited set of models; it is unclear how the approach behaves on longer tasks, other languages, or real-world production environments with codebases of millions of lines. Also, “open weights” does not equal “open source”; the models tested are not necessarily freely available. Nonetheless, the central point stands: engineering the harness is no longer plumbing work, it is research.