Nvidia shows harness matters more than model in long-term tasks

Nvidia researchers published a study over the weekend showing that the software layer that wraps a model—referred to as the harness—determines performance on long-term tasks far more than the model itself. The harness is a software layer that manages memory, context, tools and operating rules, turning a raw model into an autonomous agent. According to Adel El Hallak, vice president of product in Nvidia’s AI unit, “the industry tends to treat the agent as the model’s API, but in practice the agent consists of the model, the scaffolding around it, the runtime and the skill libraries available to it.”

The team evaluated Claude Opus 5 on the ARC-AGI-3 benchmark, a 2-D game suite without instructions that requires the model to learn the rules and win like a human. Without a tailored harness, Opus 5 achieved 30%, the highest score among all models tested. When a harness specifically designed for memory management and including a “supervisor” component was applied, the same model reached 100%, a score equivalent to human performance. The benchmark notably upset OpenAI, whose models scored less than 10% on the original test.

OpenAI conducted its own research last month and found that altering two harness settings triples scores, yet no model approached 100%. Nvidia’s approach added a supervisory agent that functions like a CEO: it pushes the primary agent when it stalls, blocks entry into dead ends, and redirects it to neglected pathways. The harness used in the experiment is called Agentic Variation Operators (AVO). It is not a brand-new Nvidia product; the company releases open components for building harnesses under the Nemo brand, with some offerings commercial.

Long-term tasks require chaining dozens of decisions over days, unlike one-off prompt responses. Microsoft research from April showed that 19 models, including frontier models, produced documents with persistent editing errors in extended work—a mistake that would lead to immediate termination for humans. Autonomous agents have already been observed deleting files, wiping entire databases, and even engaging in criminal behavior such as hacking or extortion to meet objectives. The practical takeaway is that developers building on a single-layer harness—such as Claude Code, Codex or Hermes—miss the component that separates a toy from a hired system.