Researchers from Hugging Face have introduced xHC (Expanded Hyper-Connections), a novel method that pushes the residual stream expansion of Transformers beyond the previous limit of N=4 parallel streams. The work addresses fundamental bottlenecks that prevented earlier Hyper-Connection (HC) methods from scaling effectively.
The standard Hyper-Connections expand the residual stream into N parallel streams, offering a form of memory scaling. However, prior Manifold-Constrained HC (mHC) showed diminishing returns and rapidly increasing training costs beyond N=4. The authors identify two key bottlenecks: insufficient write-back information for many streams, and cubic scaling of residual-mixing generation cost with N.
xHC introduces two innovations: temporal feature augmentation for richer write-back, and a sparse residual-stream architecture that updates only k=4 of N=16 streams while retaining dense access to the full residual state. This design enables meaningful expansion beyond N=4 for the first time in the HC family.
Experiments on 18B and 28B MoE models show strong gains. On the 18B model, xHC lowers the final training loss from 1.776 to 1.758 and raises the average downstream score from 44.8 to 48.8 over DeepSeek mHC, with broad improvements across reasoning, knowledge, and code tasks, while adding only 4.1% training FLOPs over the vanilla baseline. At 28B, xHC outperforms mHC by 3.1 average points.
Scaling-law experiments reveal that vanilla and mHC require 1.50× and 1.19× the compute of xHC, respectively, to reach the same loss. To manage memory traffic from the expanded residual state, the authors also introduce xHC-Flash, which reduces per-sublayer memory traffic from 73.5C to 40C, comparable to the 34C of mHC at N=4, while preserving full xHC gains.