A new research paper from Hugging Face presents the first systematic study of massive activations (MAs) in layer-interleaved hybrid linear-attention (HLA) large language models. The authors identify two architecture-aligned morphologies: MAs consistently spike immediately before full attention layers, forming what they call pre-attention spikes (PAS), and can persist through intervening linear attention layers, giving rise to inter-spike plateaus (ISP).
The study demonstrates that as full attention becomes denser, successive PAS become increasingly connected through ISP, ultimately recovering the stable MA morphology characteristic of full attention LLMs. This organization was found to recur across five linear attention architectures, six hybridization configurations, five data domains, and representative open-source hybrid models ranging from 1.2B to 397B parameters.
Controlled pretraining of GDN-based hybrids at scales up to 1.3B revealed that both morphologies emerge early in training and respond asymmetrically to output gating: full attention output gating strongly attenuates their absolute magnitudes without eliminating layerwise organization, whereas removing GDN gates yields comparatively modest amplification.
Mechanistically, the authors' systematic-outlier analysis supports a shared lifecycle account governed by the timing of MA cancellation. PAS follows a localized write-sink-cancel process, while the extended persistence of ISP is consistent with delayed cancellation. At the full attention limit, this account recovers the stable MA morphology seen in full attention LLMs. The code is available on GitHub.