As large language models (LLMs) handle longer contexts and more complex reasoning, the key-value (KV) cache has become a major bottleneck, consuming both memory capacity and bandwidth during token generation. High-bandwidth memory (HBM), though fast, is expensive and limited, constraining batch sizes and overall throughput. A new paper from Hugging Face researchers introduces OasisKV, a system designed to break this memory ceiling.
OasisKV takes a memory-centric approach: instead of keeping the entire KV cache in HBM, it stores the full cache in lower-cost memory tiers (such as host or remote memory) and only brings the most relevant entries into HBM for attention computation. This is possible because attention during decoding is naturally sparse—only a small subset of tokens truly matters for each step.
The key innovation is using speculative decoding to predict which tokens will be important in the next step. By drafting lookahead tokens, OasisKV can accurately identify and prefetch the necessary KV blocks ahead of time, staging them in HBM before they are needed. This lookahead prediction is so precise that it maintains accuracy within 0.7 points of full attention when operating under a 2,048-token KV budget.
Built on vLLM, OasisKV demonstrates significant throughput gains: 1.69x over dense vLLM on reasoning workloads with only 0.1 points of accuracy loss, and up to 2.1x on multi-GPU long-context serving. In prefill-decode disaggregation scenarios, it achieves about 2x dense throughput while admitting each request with 6.5–9.7x less KV and holding 2.2–2.6x less decode-node host memory than full KV transfer.
The paper concludes that OasisKV effectively expands the in-decode memory capacity by leveraging off-GPU memory, offering a practical path to scale LLM inference without being hamstrung by HBM limitations.