Linear attention layers are increasingly replacing softmax attention at scale for long-context modeling, but existing context extension methods typically apply continued pretraining without modifying these layers, overlooking the spectral properties of linear attention state dynamics. A new paper, SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization, addresses this gap by studying long-context extension of Gated DeltaNet (GDN) from the perspective of the transition matrix spectrum.
The authors identify two essential factors governing long-range information retrieval: a sufficiently broad slow spectral band aligned with the target dependency length, and the preservation of fast-decaying modes for state clearing and context switching.
Based on this observation, they propose SpectralShift, a spectral reparameterization approach for long-context continual pretraining of GDNs. The method reparameterizes the alpha projections initialization to reshape the decay spectrum by enhancing slow propagation capacity, and introduces a learning-rate scaling for alpha projections to facilitate long-context training.
Experiments show that SpectralShift consistently improves long-context capabilities over training, offering an effective and efficient solution for extending context windows of linear attention models. The code has been open-sourced at https://github.com/RUCAIBox/GDN-SpectralShift.