A newly posted research paper introduces IdeaAMBIG, a benchmark designed to measure whether research-method specifications contain enough methodological detail for a competent implementer or coding agent to build the intended method without unsupported assumptions. The authors frame this as the "codification readiness" of implementation-facing specifications.
The benchmark contains 660 evidence-grounded instances, drawn from papers, codebases, issue threads, and reproduction artifacts. Of these, 163 are real-world gaps sourced from reproducibility reports and GitHub issues, while 497 are controlled synthetic gaps injected into codification-ready references.
IdeaAMBIG evaluates three capabilities: codification-readiness assessment, defect localization, and clarification action generation. Notably, defect localization receives only the specification, whereas clarification additionally receives the annotated defect.
Across 13 LLMs, the best model achieved a 9.6% Macro Defect Recovery Rate on real-world instances, but an 80.6% Macro Clarification Action Success Rate when the defect was provided. In an oracle study, supplying the gold resolution raised the downstream codification-ready rate from 14% to 98%.
The authors conclude that defect localization is the primary bottleneck across all evaluated models, with clarification performing much better once the defect is known. They suggest that before asking models to implement a research idea, the community may first need them to reliably recognize what is missing, ambiguous, or inconsistent.