Retrieval-augmented generation (RAG) has become a cornerstone of modern AI systems, but choosing the right retrieval paradigm—lexical, dense, graph-based, or agentic—remains a challenge. A new study from Hugging Face researchers provides a systematic comparison across corpus sizes, revealing that the best approach depends heavily on scale.
The study, titled "BM25 Wins at Scale," evaluates four RAG paradigms across 28 nested corpus tiers spanning a 450-fold range, using a single reader model and judging protocol. The researchers measured accuracy, token usage, and latency while keeping questions and a fixed set of relevant and adversarial documents unchanged.
Results show a clear crossover: the File-System Agent, an agentic approach, leads at the smallest corpus sizes but becomes inefficient as the search space grows, consuming 39 times more query tokens at the largest scale. Around 10 million corpus tokens, BM25—a classical lexical retrieval method—overtakes the agent and maintains its lead at every larger tier, with an accuracy margin approaching 20 points at full scale.
Dense retrieval remains efficient but less accurate, while graph-based RAG faces construction bottlenecks before reaching deployment scale. The study concludes that corpus growth favors global candidate ranking, making lexical retrieval the strongest scalable default, while agentic reasoning is best applied after ranked discovery rather than as a replacement for it.