Should you replace your text-embedding pipeline with a large language model? A new research paper from Hugging Face researchers tackles this question with a comprehensive, cost-aware comparison. The study evaluates ten LLMs from six families and 26 embedding models (ranging from 118M to 14B parameters) across 37 tasks, including classification, semantic textual similarity (STS), clustering, pair classification, and retrieval.
The headline finding: in aggregate, the two paradigms are effectively tied. The best LLM (Gemini 3.1 Pro) scores 77.6, while the best embedding model scores 77.2—a difference of just 0.4 points. However, their strengths diverge by task type. LLMs lead on reasoning-heavy retrieval, embedding models dominate classification, and the two are evenly matched on clustering, STS, and pair classification.
The cost of that parity is staggering. An LLM can cost up to 1,431x more than an embedding model of comparable quality—USD 154 versus USD 0.11 per benchmark pass. On the same GPU, the open LLMs tested process tokens 2.5 to 736x more slowly. Reasoning tokens account for 28% to 81% of LLM inference cost, and the study's ablation shows that reducing reasoning budgets can preserve or even improve retrieval quality for most models.
The Pareto frontier—the set of optimal cost-quality trade-offs—includes the leading embedding models and only one LLM: Gemini 3.1 Pro. The authors conclude that a division of labor is optimal: use embedding models for similarity, classification, and clustering, and reserve LLMs for reasoning-intensive retrieval. Their code, datasets, and results are publicly available on GitHub.