Meta releases model that predicts which experiments deserve GPU time

A research team from Meta FAIR, Oxford and USC has built a system that ranks experiments before they run, saving roughly three hours per compute day. The problem is familiar to anyone operating a research agent: the agent proposes dozens of candidates, yet each execution consumes hours to days on H100 or H200 GPUs. Generation is cheap, validation is expensive, and the decision of what to run is the real lever on progress velocity.

The system, called Research Preference Models (RPMs), plugs into AIRA-dojo, an evolutionary tree search that spawns children through three operations: Draft, Improve, Debug. Instead of running a single child and hoping for the best, the agent invokes the operator 15 times in parallel, collects 15 unexecuted candidates, and pits them against each other in a pairwise knockout tournament. Only the winner proceeds to execution. Every comparison draws on pairs gathered during a BFS scan of the tree, each pair accompanied by the validation score it earned.

Two versions, two budgets. The first, Inference-only, is an LLM-as-a-judge that receives plans, code and search history. Its prompt was optimized with MIPROv2 from DSPy and converged on a lab-manager rubric: tolerate fixable bugs, reward extensibility, penalize duplication. Offline accuracy landed at 57.7%–59.0%. The second, Agentic, adds a sandbox that replicates the environment on a single H200, runs small pilot experiments, and decides whether to continue or stop. Two pivotal design choices shape its behavior: the remaining budget is reported as inflated (2,700 seconds versus 300 actual) so the agent does not halt prematurely, and pilots are capped at 30 with a 60-second ceiling. Because pilot time competes with the agent's own clock, the agentic selector runs only during Draft and Improve; in Debug the system falls back to random selection.

The numbers speak for themselves. The AIRS-Bench benchmark comprised 20 text and table tasks, 24 hours on a single H200 per task, 10 seeds, and Qwen3.6-27B as the backbone for both operators and the RPM — meaning the gain comes solely from the selection layer. Probability of improvement over a no-RPM baseline stood at 0.5923 and 0.5913, with 95% CI lower bounds of 0.5066 and 0.5018. But efficiency is the practical result: Inference-only reached the baseline's final score (0.684) in 14.88 hours (1.61× speedup), Agentic in 15.50 hours (1.55×). Self-hosted inference adds 0.66 hour per run; after adjustment the system still achieves 0.708 in 23.34 hours. Two new SOTA results were reported: WinoGrande 94.1% with Agentic RPM versus 90.4% for AIRA₂, and SVAMP 95.7% with Inference-only versus 94.2% for humans.

Open code, no fine-tuning. Both versions run on frozen LLMs without any fine-tuning. The Qwen3.6-27B backbone is open-weights, and the AIRA-dojo scaffold and AIRS-Bench benchmark have also been released open-source. That means any team running research agents can drop this selection layer onto their existing stack without training a new model and without paying for proprietary weights.