Linear attention replaces the growing key-value cache of standard transformers with fixed-size recurrent states, but those persistent states can themselves become a significant memory bottleneck when many requests are served concurrently. The paper STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization examines why naively quantizing these states to low precision tends to cause severe accuracy loss: quantization errors propagate through successive state updates.
The authors argue that the impact of such errors has two complementary dimensions. Temporally, errors in long-lived memory can persist across many decoding steps. Spatially, errors in different key rows affect model outputs differently, and state magnitudes vary substantially along both rows and columns.
Based on these observations, the team proposes STEPQuant, a spatial-temporal post-training quantization framework for Delta-rule recurrent states. It allocates precision according to error magnitude and memory lifetime, and jointly fits key-row and value-column scales using state distributions together with each key row's impact on output error.
Experiments on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct, across both long- and short-generation benchmarks, show STEPQuant closely matching FP32-state accuracy under a nominal 6-bit budget, while outperforming uniform INT8 in its 4-bit configuration. Integrated into SGLang with optimized GPU kernels, the 6-bit configuration achieves over 5x recurrent-state compression and reduces total serving memory by up to 68.7%.
The authors have released code at github.com/Dreamer-Toby/STEPQuant, with the paper available on arXiv.