In a new research paper, Hugging Face and collaborators present LatentPress, a novel approach to context compression that moves beyond traditional text summaries or OCR-based image rendering. Instead of reconstructing compressed context into human-readable forms, LatentPress writes conversational histories and long documents into continuous memory tokens that a frozen decoder reads directly through its input-embedding interface—eliminating text reconstruction at inference time.
The method employs a small writer model, matched to the reader, that compresses context by 4–16 times while training only an adapter of 4.2M–26.2M parameters—roughly 0.1% of the decoder's size. This lightweight design enables significant efficiency gains: writing takes about 43ms per conversation, nearly an order of magnitude faster than text summarization or OCR reconstruction, and reading is 5–9 times faster than processing raw context or cached OCR.
On the LongMemEval benchmark, LatentPress achieves 0.504 accuracy at 7.70× compression, outperforming uncompressed evidence (0.490) and far surpassing text summaries (0.184) and OCR-based compression (0.426–0.312). On LongBench-QA, in-domain writers match or exceed raw-context reading at 4–8× compression, while 16× compression trails raw context.
The researchers validate the interface under two transfer settings: zero-shot from UltraChat to LongMemEval memory QA, and from LongMemEval-derived QA to unseen LongBench document domains. These results establish direct soft tokens as a practical machine-facing context interface beyond text and vision. The implementation is available on GitHub.