A new research paper from Hugging Face introduces MirageBench, a benchmark designed to measure over-inference (OI) in personalized LLMs—the tendency of models to fabricate user attributes beyond what the evidence supports. The study, which evaluates 12 models across 7 families on 143,616 judged claims, finds that over-inference is pervasive: every model over-infers between 35% and 49% of its claims, with a cross-model mean of 41.6%.
The benchmark comprises 150 personas balanced across stereotypical, counter-stereotypical, and neutral profiles, and spans 6 personalization tasks along an "imagination gradient." A four-way faithfulness taxonomy is operationalized by an independent judge, validated against blind human annotation on 400 claims with high agreement (Cohen's kappa = 0.863 four-class, 0.900 binary).
Most strikingly, the authors report a Self-Monitoring Inversion: at the model-selection level, models' self-assessed OI is negatively rank-correlated with judge-measured OI (rho = -0.60, p = 0.044). In other words, models that claim to over-infer the least are often the ones that fabricate the most. This suggests that self-reported confidence is a misleading signal for comparing models, even though within a single model, self-audit still ranks that model's own claims moderately well (AUROC 0.58–0.83).
The study also shows that OI is task-dependent, ranging from 27% to 59%, and that in a multi-turn pilot, inferred attributes accumulate approximately linearly with little revision. The authors argue that external verification, rather than model self-report, is a more reliable foundation for trustworthy personalization.