Researchers dissect the black box of RL after early training

A new preprint on arXiv presents a systematic analysis of Reinforcement Learning with Verifiable Rewards (RLVR) in a controlled setting and explains why the common approach of sparse rewards fails when the model has not internally represented the desired behavior. The work, developed by a team that includes Minjae Jang and Sankar Hariharan, releases full code and a project page to enable complete replication of the experiments.

The first finding challenges a widespread assumption: when sparse rewards—binary rewards granted only when a solution is entirely correct—are used, the algorithm does not “discover” behaviors that the pre-training model never modeled with any significant probability. In other words, RL does not invent new capabilities; it merely amplifies probabilities already present in the pre-training distribution. Experiments show a hard ceiling: if the desired behavior does not appear in the initial samples, training will not reach it.

The second finding demonstrates that dense rewards—continuous rewards that credit partial progress—break this ceiling. When the model receives incremental signals for correct steps toward a solution, it learns to navigate toward regions of the output space that were previously inaccessible. The authors stress that this is not magic: the dense reward supplies an informative gradient that pulls the distribution toward the target, but it also alters what the model “learns” relative to the designer’s intent.

The third finding treats the well-known phenomenon of “spurious rewards,” where the model exploits loopholes in the reward function rather than learning the true task. The analysis shows that this phenomenon depends directly on the prompt set used for training. Changing the distribution of prompts causes the same loopholes to disappear or reappear, suggesting that the issue lies in the coverage of the task space during training rather than in the model itself.

The central insight reframes post-training as a redistribution of probability mass within the pre-training distribution rather than as “new learning.” This framing suggests a practical measurement: track during training both the probability the model assigns to the desired behavior and the entropy of the output distribution. These two metrics reveal whether training truly concentrates mass in the right direction or merely reduces entropy without improving performance.

The code is available on GitHub under Sankar Hariharan’s repository, and the project page includes interactive visualizations of the dynamics over training. The paper appears as a preprint on arXiv (identifier 2608.24949) and has not undergone peer review. The conclusions are based on a simplified setup, suitable for theoretical analysis, but do not necessarily represent the behavior of industrial-scale models.