Quantization is a standard tool for reducing the cost of training and running large language models, but pushing the attention module to very low bit-widths tends to introduce large errors and degrade performance. Most existing approaches rely on smoothing techniques to handle outliers. A new paper, HyQuant: Hybrid-Precision Quantization for LLM Attention, takes a different route: a hybrid design that keeps a small, accuracy-critical subset of attention states in high precision while quantizing the rest.
The method is motivated by an observed pattern in attention maps across Qwen3, Llama-3, Gemma 4 and Qwen3.5: persistent "vertical lines" where the top-5% key positions plus a 128-token local window capture 82–86% of the attention mass. HyQuant identifies these regions using lightweight vertical-line-aware attention-pattern signals, then keeps the vertical-line tokens and the local window in FP16 while quantizing the remaining context to 4-bit (K4V4).
In the prefill stage, HyQuant uses a hybrid-precision quantized attention operator that preserves the vertical-line tokens and a local sliding window in full precision. In the decode stage, the same principle is applied to KV-cache compression, with KV dequantization fused into the attention computation to improve memory and hardware efficiency. According to the authors, identifying the vertical lines costs only 3–5% of runtime, and the fused kernels avoid materializing the KV cache.
Reported results are strong: the decode kernel runs up to 3.58x faster than FlashAttention-2 at 32K context, with end-to-end decode 1.04–1.17x faster, while KIVI and KVTuner end up slower than FA2 (0.69–0.80x). On LongBench with Qwen3-8B in thinking mode, HyQuant averages 45.04 versus 44.59 for FA2, compared with 37.7–40.5 for KIVI, SageAttention and KVTuner; the authors say results hold on Qwen3-32B, Llama-3.1-8B and GLM-4-9B. HyQuant is also the only method tested that still runs at batch 16 with a 32K prefix (231.6 tok/s), where FA2, KIVI and KVTuner run out of memory.
The work is accepted to EMNLP 2026 Main, and code is available at github.com/jerrysfls/HyQuant.