Relative Scaling Laws for LLMs

Abstract

Scaling laws describe how language models improve with additional data, parameters, and compute. While widely used, they are typically measured on aggregate test sets. Aggregate evaluations yield clean trends but average over heterogeneous subpopulations, obscuring performance disparities. We introduce \textbf{relative scaling laws}, which track how performance gaps between test distributions evolve with scale rather than focusing solely on absolute error. Using 255 decoder-only Transformers trained under matched-compute (\emph{IsoFLOP}) budgets from $3\times10^{18}$--$3\times10^{20}$ FLOPs on standard pretraining datasets, we find diverse trajectories: academic domains on MMLU converge toward parity; regional English dialects shift depending on population size; and preference-based AI risk indicators split, with capability- and influence-related indicators increasing during pretraining while adversarial indicators do not. These results show that although scaling improves overall performance, it is not a universal equalizer. To support further study, we release all model checkpoints from this work to enable practitioners to measure \emph{relative} alongside traditional scaling laws, in order to better prioritize robustness challenges in light of the bitter lesson.