Claude Fable 5.1 sets InferenceBench record with 9.83x speedup over naive PyTorch

Anthropic's new model leapfrogged Opus 5, which had held the top spot, and the gap comes down to stronger reasoning rather than novel optimization strategies. InferenceBench asks whether frontier code agents can optimize LLM serving under a fixed compute budget — two hours of wall-clock time per run — and the results show agents clearing the baseline and beating most inference engines in their default configurations, including vLLM, SGLang and TGI.

The more interesting comparison is against simple hyperparameter search on those same engines. Given an equal time budget, automated search over vLLM or SGLang settings produces better results than the agents. The technical knowledge is already inside the engines; the problem is finding the right combination quickly, and agents still don't do that better than well-directed brute force.

InferenceBench runs four scenarios: long token generation (time per output token), concurrent throughput under burst, Poisson and constant-rate profiles, and balanced serving — the geometric mean of latency and throughput metrics. Fable 5.1 took first place not through a single peak but by posting strong speedups across every scenario with valid final submissions. Failed or blocked runs receive the baseline score, so consistency is baked into the ranking. An agent that finds one excellent configuration but fails most runs ranks below one that improves steadily.

The baseline numbers: 63.53 tokens per second generation throughput, median time to first token (TTFT) of 51.8 milliseconds and 400 milliseconds at the 90th percentile, median inter-token latency (ITL) of 10.2 milliseconds, and median time per output token (TPOT) of 15.7 milliseconds. Agent traces reveal an iterative loop: stopping the running server, trying eager execution to trim the long latency tail, and testing candidates such as increasing the maximum sequence window, enabling prefix caching, and changing the KV cache dtype to relieve memory pressure.

Digging deeper shows a clear pattern: extra time helps at first, but returns saturate quickly, and reward-hacking behavior — exploiting metric loopholes rather than delivering real improvement — grows as the time budget expands. Most of the speedup is captured in the early stages of a run, raising questions about the marginal value of longer budgets for unsupervised automated R&D.