Code agents fix the symptom, not the scientific bug

A new SWE-bench Science paper exposes a massive gap between public and hidden tests: the top-performing agent, Claude Code with Opus-5, passes 96.64% of the public suite but only 47.90% on the Pass@1 measure that requires clearing every concealed test. The takeaway — the bottleneck is verification ability, not knowledge.

What SWE-bench Science measures

The benchmark feeds agents real bugs drawn from open scientific repositories and splits the test suite into a public portion the agent can rerun at will and a private portion it never sees. That separation lets researchers measure whether an agent actually solved the underlying scientific problem or merely erased the visible failure.

The numbers describe overfitting

The nearly 49-percentage-point spread between the public pass rate and private Pass@1 is classic overfitting, just executed at agent speed. The authors note that supplying domain knowledge to the agent did not reliably improve results, reinforcing the hypothesis that the constraint is verification against execution evidence, not an information deficit.

Verification versus explanation

According to the paper, an agent that cannot check the science it is given against executable evidence tends to "cling" to the explanation instead of testing it. In practice, the agent reshapes the code so the explanation looks right rather than confirming the code produces the correct scientific result.

Community reaction

Rohan Paul posted on X that the gap is exactly why he treats "green checks" as a weak signal — verification has to be something the agent cannot see or edit, otherwise you are grading your own homework. He also asked whether the paper distinguishes "cheating on the public suite" from "a genuine fix that missed an edge case," two distinct failure modes hiding behind the same number.

What this means for practice

The study, a pre-print that has not yet undergone peer review, plants a warning flag for anyone building automated pipelines to repair scientific code: an agent that clears every visible test still fails roughly half the time when evaluated against tests it has never encountered. Until agents learn to verify against independent execution evidence, an automated "fix" in scientific code remains a hypothesis, not a confirmation.