Researchers from Hugging Face have published a critical analysis of contamination in multimodal automated fact-checking (MAFC) benchmarks, accepted at ACM MM 2026. The paper, titled "Novel Claim or Déjà Vu? Rethinking 'Contamination-Free' Dynamic Evaluation for Multimodal Automated Fact-Checking," challenges the assumption that dynamic benchmarks—which use claims published after an LLM's knowledge cutoff—are contamination-free.
The team constructed a new dynamic benchmark, ClaimReview2025Q4, and compared it with the static AVeriTeC benchmark. They developed a Contamination Detection Pipeline to quantify knowledge contamination risk in LLMs and VLMs. Their experiments yielded 16 findings, with three key results: (1) dynamic evaluation reduces but does not eliminate contamination—17.09% to 29.30% of post-cutoff claims remain potentially contaminated; (2) many new claims can be verified using pre-cutoff public knowledge, either directly or by synthesizing multiple pieces; and (3) contamination can inflate Macro-F1 by up to 11.34 points and distort system rankings.
The study provides practical guidelines for trustworthy MAFC evaluation, emphasizing that high benchmark scores do not necessarily reflect real-world fact-checking ability. The authors call for rethinking evaluation practices to focus on true unseen claims.