Researchers have proposed SparseDecoding, a pruning framework designed specifically for the decoding stage of large language model inference, where memory-bound behavior drives significant latency. The work is presented as a research paper on Hugging Face.
The authors identify two shortcomings in existing approaches. First, layer-wise training-free pruning methods guided by the Hessian typically compute that Hessian from pre-collected natural text sequences, while during decoding the model consumes its own generated tokens. This creates a distribution shift: the Hessian computed on natural sequences differs from the one that would be computed on generated sequences, pushing the activation distribution during generation away from the distribution used for pruning and degrading the pruned model's quality.
Second, most existing pruning methods that achieve real speedups are built around sparse matrix-matrix (SpMM) multiplication, offering limited support for the sparse matrix-vector (SpMV) operations that dominate decoding.
SparseDecoding addresses both issues on two axes. Algorithmically, it builds calibration matrices from layer-wise activations collected during the dense model's autoregressive generation, excluding the prefill phase, so the pruning objective aligns with decoding activations. On the systems side, it introduces an optimized N:M sparse matrix-vector kernel using bitmask indexing and fixed-step traversal.
Experiments on representative models — Llama-3.1-8B, Llama-3.3-70B, and Qwen3-14B / 32B — show the method consistently outperforms standard fixed-text calibration on long-form generation benchmarks, while achieving up to 1.48x end-to-end wall-clock decoding speedup on A100 GPUs.