Large language models are increasingly used in scientific evaluation, whether as automated reviewers or as assistants to human reviewers. As model-generated reviews spread into public data and future training corpora, peer review can become recursive: later reviewers learn from judgments produced by earlier models.
A new paper on Hugging Face studies one step of this feedback loop in a controlled setting. Starting from Llama 3.1 8B, the researchers first fine-tuned a reviewer on official ICLR reviews from 2018–2023, then trained four successor models on ICLR 2024 data using systematically varied mixtures of official and model-generated reviews.
The study reports that introducing synthetic reviews compresses rating distributions and reduces both same-paper and corpus-level semantic diversity. The authors name this pattern scientific-judgment collapse.
To address the failure mode, the team introduces TrustReviewer, an open-source LLM-based system for generating peer reviews of AI and machine learning papers. It intervenes at two stages: for training-time prevention, the core reviewer is trained in a single stage on a curated corpus designed to reduce low-quality and semantically degenerate supervision; for test-time correction, paired activation steering aims to further mitigate residual tendencies toward collapsed judgments without additional training or expert annotation.
The authors say the results characterize a concrete risk of recursive reviewer training and offer practical interventions for preserving judgment diversity and improving recommendation alignment in AI-assisted scientific evaluation.