HarnessDev shows only 34 of 64 harness evolutions generalize

A research team from ByteDance Seed, Singapore's SUTD, Georgia Tech, M-A-P and TokenWave.AI has released HarnessDev, a benchmark that flips the evaluation script: instead of measuring a model's answer, it measures the agent harness the model writes for itself. The harness is the loop that runs the model, tools, context management, recovery and verification; differences between harnesses explain why GPT-5 solves 35.2% of tasks on Terminus 2 but 49.6% on Codex CLI with the same weights.

Six models — Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max and Seed 2.0 Pro — received an identical weak seed (primitive file, search and process operations, no loop, planner or verifier) and were asked to build a full harness for each of five benchmarks across four domains: code (SWE-bench Pro, Terminal-Bench 2.1), search (BrowseComp), writing (EQ-Bench3) and ML experiments (MLE-bench). Total: 2,207 tasks. Each model built three harnesses per benchmark (avg@3). Evaluation ran two ways: Self-Eval, where the author model also executes, and Unified-Eval, where all harnesses run under Gemini 3.1 Pro as executor.

Opus leads on average, but gaps depend on domain. In Self-Eval Opus 4.8 scored 67.8 average against a human reference of 86.2. On code: Opus 69.3 on SWE-Pro versus 80.0 reference; Gemini led Terminal-Bench at 68.8 versus 88.8. Search showed the widest gap: GPT-5.5 reached 52.6 on BrowseComp versus 92.2 reference. In writing Opus beat the reference (84.6 vs 83.7), and in ML both Opus (32.9) and Gemini (32.4) cleared the 24.0 human reference. Code, terminal and search references come from OpenAI's GPT-5.6 report, not fresh runs.

Code volume does not predict quality, and much code is dead. The 18 harnesses added 17,111 net lines, yet Gemini added the fewest (1,006 lines) and led on terminal. Self-test count barely correlated with score (Spearman 0.13–0.26); revision-call count correlated better (0.57). Of 108 generated components, 72 were actually invoked and 18 never fired — all state and memory management. 11 of 18 harnesses defined a State class, but zero checkpoint events appeared across 26,679 trajectories. 124 of 587 write features were dead code.

Swapping the executor reshuffles rankings, sometimes breaking them. On MLE-bench token consumption varied 19×: GPT-5.5 earned 19.1% medals with 29.3 million tokens, DeepSeek V4 reached 19.6% with 208.4 million. Under the Gemini executor, Qwen gained 17.6 points on BrowseComp and 12.9 on MLE-bench, but Opus collapsed on SWE-Pro from 69.3 to 33.0 — partly because one Opus harness hard-coded a 120-step limit around the original executor. In Opus's search harness, duplicate-query rate jumped from 10.1% to 88.2% after the swap.

Evolution: 9 lineages, 73 official versions, 34 changes that held. The evolution phase let each author repair its own code against 100 fixed SWE-bench Pro tasks and all 89 Terminal-Bench 2.1 tasks, with a budget of 10 official eval pairs and up to two five-task probes between pairs. 9 lineages (five self-runtime, four with fixed Gemini) produced 73 official versions and 64 adjacent swaps, but only 34 of those swaps also improved on the 630 held-out tasks the author never saw. The takeaway: models can write working harnesses, but most of their directed-evolution attempts don't survive outside the training set.