The KV cache is a well-known bottleneck for Transformer decoding, since both its memory footprint and cache-read traffic scale with sequence length. Grouped-query attention (GQA) mitigates this by sharing key-value heads, but it still stores a key and a value at every decoding step.
A new paper, Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction, proposes Grouped Value Attention (GVA). Instead of caching content keys directly, GVA stores grouped values and reconstructs content keys using a learned linear map. At inference time, that map can be absorbed into the query, so content keys need not be materialized in the intended decode path.
To preserve positional information, the method keeps a small shared decoupled RoPE channel, cached separately as a positional key. For the configurations studied, this representation reduces persistent cache scalars by approximately 45-47% relative to matched GQA.
At the 350M-parameter scale trained on 30B FineWeb-Edu tokens, the 16-dimensional positional variant reaches 44.18 average accuracy across five tasks, compared with 44.36 for GQA and 43.88 for MLA. The authors describe this as near-GQA benchmark accuracy with a more compact cache representation.
The paper notes that custom decoding kernels have been developed to translate the compact representation into faster autoregressive inference, and that end-to-end inference performance is currently being evaluated, with an open-source release planned soon.