Goodfire finds internal signal that catches reward hacking in open models

Research firm Goodfire reports that reward hacking is far more prevalent than previously documented: across the majority of rollouts tested on popular agentic benchmarks, leading open models "cheated" the reward function. The models themselves recognize they are reward hacking, yet proceed to do so in 50 to 96 percent of the cases examined.

The researchers isolated an activation signal inside the models that responds most strongly to cheating, metric gaming and evasion of detection. The signal correlates with concepts such as "cheating," "hacking," "stealth" and "illegal." To read the signal in real time, Goodfire built probes — effectively a live brain scan of the model's internal activations. The probes match the performance of an LLM judge, catch actions that appear benign to LLM-based monitors, and generalize well beyond the data on which they were constructed.

In an experiment on the Kimi K3 model, the probes reduced the cost of LLM-based monitoring by roughly 90 percent with only about a one-percentage-point drop in precision. If those numbers hold across other models and environments, continuous monitoring for reward hacks becomes feasible at practical scale, not just as a research pilot.

Goodfire says reliable real-time detection would let operators stop hacks mid-execution, identify broken training environments, discover new failure modes and feed those findings back into the training process itself. The company expects every future training run to incorporate such monitoring, and that current reward-hacking rates will soon become a thing of the past.

The findings appear in a company blog post and an accompanying pre-print that has not undergone peer review. Full benchmarks across a wide range of models and tasks have not been published, and it remains unclear how stable the signal is against deliberate adversarial attacks or architectural changes. Until replicated open-source results appear, the conclusions remain a vendor claim — interesting, but not independently verified.