Speculative decoding (SD) is a technique that accelerates large language model inference by using a lightweight draft model to propose tokens, which are then verified in parallel by a larger target model. Recent advances have introduced 'lossy verification' schemes that relax strict distributional matching to further improve efficiency. However, this relaxation can silently alter the decoding distribution, leading to unstable or severely degraded generation quality.
A new paper from Hugging Face, titled 'Revisiting Lossy Verification in Speculative Decoding: Mechanisms, Trade-offs, and Failure Modes,' offers a principled analysis of the distributions induced by these methods. The authors show that many seemingly distinct approaches can be classified into two categories: truncation-based verification and collaborative verification. This classification reveals that differences between methods are often superficial.
The paper also constructs a diagnostic evaluation framework across curated benchmarks. For truncation-based methods, the researchers identify a fundamental pitfall: performance can degrade significantly compared to the true truncation sampling baseline due to distributional distortion. For collaborative verification, they uncover a key principle: controlling the overshoot of draft probabilities relative to target probabilities is essential to prevent low-quality outputs.
The authors provide a 'Principled Characterization' of lossy verification, revealing its underlying paradigms and distributional failure modes. The code is available on GitHub at https://github.com/ZhouYuxuanYX/Fast-HSD.