New benchmark tests whether agents survive hours of autonomous work

Matt Stallone announced SWE Odyssey, a standard benchmark in the ultra-long-term evaluation family intended to answer a simple question: can agents work autonomously for hours and still build the right product. The announcement comes amid a growing saturation of existing benchmarks that cease to differentiate the most advanced models.

57 tasks across five genres with context of tens of millions of tokens

The benchmark comprises 57 tasks spread over five genres: building products from scratch (in the style of Slack), fixing deeply rooted bugs, performance optimization, and translating entire systems between languages (think rewriting to Bun). The average token count per task ranges from 30 million to more, a scale that forces the agent to maintain context over an exceptionally long period.

Evaluation method fundamentally different: no LLM judge and no hidden test suites

The biggest change lies in measuring correctness. SWE Odyssey does not use an LLM as a code judge and does not rely on hidden test suites. Instead, the system inspects the delivered product through deterministic behavioral observers. If the product requires registration, the observer checks the actual registration flow; if collaborative editing is required, the observer verifies that it works in real time.

Harness-based goals made agents do more work, but not always in the right direction

The data revealed an interesting pattern: when tasks were defined via a /goal harness-based approach, agents performed more work, but a large portion of it was tangential to the specification. GPT-5.6-Sol was observed entering loops of feature implementation, writing additional tests, then re-implementing against the same tests—excess effort that did not advance the required outcome.

Dramatic efficiency gap between leading models

Traces exposed a significant efficiency gap: Claude-Sonnet-5 consumed more than double the tokens compared to GPT-5.6-Sol, yet could not break out of its performance band. In ultra-long horizons, autonomy proves to be about preserving intent, planning the right tasks in the right order, and avoiding unnecessary work, not merely the number of tokens burned.

Correctness validation through two reference implementations and targeted mutations

To certify correctness, each observed specification was checked against two independently built reference implementations and against a full set of targeted mutations that demonstrate every behavior is both realizable and meaningfully enforced during scoring. While agents explored the virtual environment...