A newly posted research paper on Hugging Face proposes disaggregated quantization (DQ), an approach that treats the two phases of large language model inference — prefill and decode — as distinct workloads deserving their own quantization strategies. The authors argue that prefill and decode reward different trade-offs: low-precision arithmetic speeds up prompt processing, while compact weights reduce memory traffic during token generation.
Under DQ, computation formats, weights, and storage placement are specialized per phase. The paper reports that removing activation quantization specifically on decode improves accuracy on decode-heavy tasks without raising inference cost. Separately, training compute-native prefill weights accelerates prompt processing compared with weight-only inference while matching or exceeding its accuracy at 2–3-bit decode on both decode-heavy and prefill-heavy tasks.
The authors demonstrate the idea on Qwen 3 and Gemma 3. Using released Qwen3.8-27B GGUF decoders, they trained an NVFP4 prefiller that improved 1-bit accuracy by 32.5 points on MMLU-Pro and 35.3 points on MMMU-Pro, without modifying the decode checkpoint.
To fit the additional checkpoint on a single device, the paper introduces offloaded disaggregated prefill (ODP), which streams prefill weights from SSD and amortizes loading over prompt length. On the same 27B model, ODP reportedly delivers a 1.78x time-to-first-token speedup over the weight-only baseline at 8K prompt length in llama.cpp.
The work evaluates accuracy under disaggregated serving in vLLM and further validates shared-weight format disaggregation through post-training quantization on models up to 2.8 trillion parameters. Code and an NVFP4 prefiller checkpoint have been released alongside the paper.