WeaveBench records only 41.2% success rate, the highest recorded to date

The reliability of long-horizon agents remains far from achieved, despite model improvements. On the WeaveBench benchmark, which comprises 114 hybrid tasks that combine a graphical interface and a command line, the official highest success rate stands at 41.2% only, indicating a significant gap before the technology can be used in production. Existing models can perform local steps, but without an explicitly controlled and recorded state they lose the ability to preserve consistency across the entire task. As a result, an agent that can solve a single step may still fail when its history becomes unreliable—it does not know what was completed, what failed, and what remains.

The paper introducing LongHorizon-Harness frames the issue as a task-state problem and argues that reliability depends not only on the model itself but also on the surrounding harness. The authors claim that shifting from internal memory management to external, controlled logging of each step is the key to improving reliability.

Chamet, a lecturer at Stanford, summed up the situation in direct terms: “Long-horizon tasks are still a joke. They do not work, and I do not care what anybody says. Do not show me a stupid evaluation. Do not tell me about some dumb script you ran for 48 hours. Long-horizon tasks are not handled well. They simply do not work.”

The metric remains low.