Neural scaling laws are essential for developing large language models, but standard formulations often fail to predict loss accurately when data is scarce or when models are overtrained. This inaccuracy stems from a core assumption that model size and training data affect loss independently.
To address this, researchers from Hugging Face introduce the Skaling law, a generalized functional form that couples model capacity and data through a single interaction exponent. This simple extension significantly improves loss prediction across both interpolation and extrapolation regimes.
In experiments, the Skaling law reduced the Mean Absolute Percentage Error (MAPE) by 1.5-3x compared to standard scaling laws. When combined with a sparse grid strategy that focuses on low-compute regimes, it achieved accurate full-grid extrapolation using approximately 10x less compute than uniform sweeps.
This approach enables reliable performance prediction from small-scale experiments, offering a more robust and resource-efficient framework for allocating compute budgets in next-generation model training.