Researchers have introduced GeoPair, a training-free framework for compressing Transformer models by exploiting redundancies that span multiple layers rather than treating each layer in isolation. The work appears as a paper on Hugging Face.
According to the paper, existing post-training compression pipelines typically optimize layers independently or rely on heuristic grouping strategies that ignore the activation geometry specific to each layer. GeoPair instead sequentially optimizes cross-layer weight pairings alongside shared-dictionary factorizations.
The method identifies structurally compatible projections and learns a shared representation intended to better preserve each layer's distinct calibration geometry, rather than forcing adjacent layers to share a basis or heuristically merging activation statistics. When combined with structured sparsity, this produces efficient weight decompositions while maintaining functional fidelity, the authors state.
The paper reports state-of-the-art results across diverse architectures, scales, and modalities, consistently outperforming independent structured weight decompositions and alternative pairwise weight factorizations that use heuristic grouping. The authors frame the approach as replacing heuristic engineering with a convergent, optimization-driven pipeline, providing a theoretically grounded foundation for scalable Transformer compression.