Factual question answering typically assumes a single canonical answer, but real-world knowledge often contains divergent accounts, especially for long-tail facts. A new benchmark, ElephantBench, aims to evaluate whether large language models (LLMs) retain multiple conflicting accounts of such facts, exposing what the authors call 'epistemic myopia' in parametric memory.
The benchmark, introduced in a paper on Hugging Face, comprises 1,094 questions generated through an auditable graph-based pipeline. The pipeline retrieves related documents from a low-exposure web corpus, identifies naturally occurring disagreements, and converts them into multi-account QA records. Each answer is verified against originating documents and authoritative public web sources, then reviewed by human annotators.
Across 32 models, even the strongest model recovers both accounts on only 52.4% of questions. On nearly all remaining questions, it recalls one account but omits the other. Scaling model size and inference-time reasoning improve recall but do not eliminate this incompleteness.
Corpus analysis shows that exposure imbalance favors the dominant account, whereas greater minority-side exposure is associated with more complete recall. These findings establish ElephantBench as a reproducible knowledge probe for diagnosing epistemic myopia in LLMs.
The authors argue that their graph-based benchmark construction pipeline provides an efficient and scalable way to turn long-tail corpora into source-traceable knowledge probes, supporting efforts to evaluate and advance the epistemic rigour of next-generation LLMs. Code is available at https://github.com/Tencent/ElephantBench.