A new research paper from Hugging Face investigates the dynamics of stochastic gradient descent (SGD), the workhorse optimization algorithm for deep learning. While it is known that SGD steers neural networks toward simpler subnetworks, the temporal progression of this collapse has remained unclear.
The authors propose modeling stochastic gradient flow as a percolation process, where architectural symmetries force subnetworks to merge in discrete simultaneous blocks rather than one at a time. These structural transitions manifest as variance spikes in a macroscopic order parameter, echoing physical phase transitions.
Importantly, the study shows that this trapping mechanism and its associated scaling cascade extend to other popular optimizers, Adam and AdamW, under an explicit heavy-tailed noise model. This suggests a universal principle governing optimization dynamics across different algorithms.
The findings offer a new lens for understanding how deep networks achieve sparse, low-rank representations, potentially informing future algorithm design and theoretical analysis.