AI-generated content — often called "AI slop" — is spreading across academia, but slop in scientific papers follows patterns that existing token-based detectors struggle to catch. A new preprint, Science or Slop?: Benchmarking and Mitigating Scientific Slop in AI-Generated Papers, argues that each section of such a paper can look plausible while the scientific reasoning connecting those sections breaks down, misleading readers about the work's quality.
The authors define this failure as scientific slop and benchmark it through six measures spanning Structure, Argument, and Artifacts. To support the evaluation, they built SciSlopBench, a dataset of 390 AI-generated papers — mostly in computer science but also covering the life, social, and natural sciences — each paired with a human-written paper matched by research problem and contribution type.
According to the paper, these measures identify the AI-written paper in each pair with 85.9% accuracy, compared with 68.7% for the detector Binoculars. The authors also report that higher scientific slop correlates with lower ICLR ratings and distinguishes rejected from accepted papers above chance in every year from 2017 to 2025.
Simply optimizing the measures is not enough, the authors note. They propose SciSlopHarness, a harness-level framework that guides a fixed LLM to revise slop only where the experiment records support the change. Standard revisions leave residual slop, and direct slop-aware prompting triggers reward hacking, but SciSlopHarness reportedly reduces the remaining AI-human gap by 63% over the strongest revision baseline without needing human reference targets.
The authors conclude that AI-generated scientific papers leave fundamental traces in their global reasoning, and that responsible mitigation requires strict evidentiary grounding rather than mere prose refinement. They also invite readers to test their own papers via a project page and to contribute new slop types for a future version, and say they plan to run the approach on 60,000 ICLR submissions.