Meta paper: small models aren't bad at predicting scale — they're just undertuned

A new Meta paper on arXiv argues that the difficulty of forecasting large-model behavior from small-scale experiments isn't a fundamental limitation of small models. It's that they're unusually sensitive to hyperparameters. The researchers found scaling laws start to appear around 4 million parameters — a size trainable in under an hour on a single GPU — but only when the hyperparameter space is swept broadly enough.

The experiments paint a clear picture. With 4 or 16 configurations per scale, the scaling law is nearly invisible. At 64 configurations it emerges but extrapolation is poor. At 256 configurations it becomes accurate. In other words, small models demand far wider hyperparameter search to reveal the true trend. That insight changes the math for researchers: instead of scaling up the model in cheap experiments, spend the same compute budget on tuning breadth.

The reason scaling laws look cleaner in large models is straightforward. As the model grows, good settings occupy a larger volume of the search space, and the number of effective hyperparameters near the optimum drops toward 1. Put simply, large models are more forgiving of imperfect tuning. As a sanity check, the small-scale runs reproduced the known result: pre-norm transformers scale better than post-norm in the tested range.

There's a limit to how far you can extrapolate. When predicting far beyond the measured size range, statistical errors take over and the forecast breaks down. The practical takeaway for model researchers: cheap experiments can work, but only if you invest in real tuning breadth — not just by shrinking the model and hoping for the best. The paper, "Small-Scale Experiments: Are They There Yet?", is available at arXiv:2608.11859.