Researchers have introduced Multi-Stream Attention Residuals (MSAR), a new residual architecture that extends the concept of Attention Residuals (AttnRes) to multi-head depth attention. The work, published as a paper on Hugging Face, frames AttnRes as single-stream softmax attention along the network depth axis and draws inspiration from the success of multi-head attention along the sequence axis.
MSAR generates stream diversity from sequence context using multi-scale causal convolutions, enabling different heads to retrieve historical representations through distinct contextual views while preserving the integrity of the hidden representation. The retrieved streams are then combined via lightweight adaptive mixing to produce the single-stream layer input.
A key efficiency feature: by retaining single-stream sources and expanding them on demand, MSAR supports multi-stream computation without requiring persistent multi-stream storage or communication, keeping overhead minimal.
Across all 12 downstream benchmarks at the 18B and 28B Mixture-of-Experts (MoE) scales, MSAR consistently outperformed prior residual variants, including standard residual connections, manifold-constrained Hyper-Connections (mHC), and AttnRes.
Scaling-law experiments showed an iso-loss compute speedup of 1.58× over the standard residual baseline, compared with 1.35× for AttnRes and 1.30× for mHC. The authors conclude that MSAR is a promising residual architecture for large-scale pre-training.