A Spectral Transport Mechanism Underlying the Training Dynamics of Large Language Models

Abstract

Large language models exhibit remarkably regular loss trajectories during training, often following approximate power-law decay over long optimization intervals. The dynamical origin of this phenomenon remains poorly understood. We derive an exact transport--dissipation equation governing the spectral distribution of prediction error under gradient flow, with the irreducible entropy cleanly separated via a KL-divergence-based error observable. The equation is exact; what requires empirical input is the behavior of the spectral flux $\mathcal{F}$ during training. Analyzing public LLM checkpoints via stochastic Lanczos probes, we find that $\mathcal{F}$ is small throughout training: approximately zero in an extended quasi-static phase, and small but nonzero and directional thereafter. Because $\mathcal{F}$ remains weak, perturbation theory yields power-law loss decay as the leading-order solution over any finite window. The first-order correction predicts a systematic, monotonically growing undershoot relative to any early power-law extrapolation. We confirm this across three Pythia scales (410M--7B), where the deviation onset aligns in token count rather than parameter count, and at the loss level on OLMo-2 (7B), which differs in architecture and data. Preliminary results further indicate that the same spectral and loss signatures appear under a Warmup-Stable-Decay schedule, with the drift onset falling at the cooldown.