A new research paper, Disentangling Representation Evolution in Transformers through Directional Decomposition, examines how Transformer representations change as they pass through a network. The authors frame this evolution as a functional geometry, splitting each learned update into two parts: a component parallel to the current representation and a component perpendicular to it.
Across pretrained models, the team reports that parallel components are widespread — appearing well beyond the residual identity path. Crucially, their functional role depends heavily on where the geometry is measured. The researchers apply the decomposition in two spaces: attention and MLP updates relative to the hidden state, and attention value aggregation relative to the current token's value.
Targeted editing experiments reveal a strong asymmetry. In value space, manipulating the parallel component while excluding the self-message is markedly more robust than comparable interventions in residual space or along perpendicular directions. This approach preserves the direct self message while scaling only the non-self aggregate.
The same decomposition offers a component-resolved view of compression-induced error. According to the paper, perpendicular error separates compression methods more clearly than parallel error, suggesting a diagnostic use for the framework.
The authors also report that suppressing the full-aggregate parallel component during from-scratch pretraining lowers validation-loss trajectories and improves downstream averages, with the value-space variant showing the strongest effect. The work connects representation geometry to editing robustness, compression diagnosis, and training-time intervention. Code is available in the project repository.