Hard prompt compression, a technique used to reduce the cost of long-context inference, typically works by scoring individual tokens, sentences, or chunks and keeping only the highest-scoring ones under a budget. However, a new research paper from Hugging Face identifies a structural flaw in this approach: independent selection can split dependent pairs of evidence, keeping one part while discarding the other. When the retained text contains an answer but the deleted text defines the entity needed to interpret it, the result is what the authors call referential dangling.
The problem is systematic. At a compression ratio of 0.30, the compressor Beaver, which ranks coherent chunks using Qwen3-0.6B embeddings, leaves the answer path incomplete in 34–54% of bridge examples across three multi-hop question answering datasets. On a shared HotpotQA bridge set, all six hard compressors tested exhibit dangling at rates up to 60%, and every document in LongBench-v2 Single-Document QA contains at least one dangling reference.
The impact is significant. On dangling examples evaluated with Qwen3-8B, reinserting the missing supporting paragraph while removing nonsupporting paragraphs to maintain the token budget improves accuracy by 29–34 percentage points (p < 0.0001), recovering at least 88% of the gap to contexts retaining both supporting paragraphs. Even stronger answer models do not absorb the loss: on MuSiQue, GPT-5.5 is 8.8 points less accurate on compressed contexts than on contexts retaining both supporting paragraphs.
The authors also propose a practical fix: a compact classifier that ranks omitted sentences by whether they are needed to interpret retained text, and reinserts the top-ranked candidates without support annotations at inference. On HotpotQA with Qwen3-8B, this automatic restoration improves accuracy by 4.7 points while changing the compression ratio only from 0.30 to 0.31. The takeaway: hard compressors should optimize for both relevance and referential completeness.