A newly posted Hugging Face paper, Periodic Weak Spots: Phase Sensitivity from Chunked KV-Cache Compression, describes a systematic asymmetry in long-context models that compress their key-value cache in fixed-stride windows. Because such compression groups consecutive tokens into fewer cache entries, each token acquires a phase — its position relative to compression-window boundaries — and the same information can be easy to retrieve at one phase and hard at another.
The authors call this periodic variation phase sensitivity. In large open-weight models using chunked compression, they report that long-context retrieval accuracy can differ by up to 40 percentage points across phases, meaning strong average benchmark scores can mask recurring positional failures.
To isolate the cause, the team pretrained a family of transformers from scratch across multiple KV-compression designs and reproduced the effect. Mechanistic analysis using causal interventions pointed to phase specialization: different attention components contribute asymmetrically to retrieval at different source phases. The paper also analyzes idealized retrieval models, suggesting gradient-flow dynamics may favor sharp specialization on a single slot.
The work concludes that evaluating models with chunked KV-cache compression requires measuring accuracy across compression phases, not just the mean, and recommends reporting long-context accuracy by phase alongside aggregate scores.