Coding agents are usually evaluated against behavior described in issues or written instructions. In real web development, however, agents often have to infer the intended behavior from working software and reproduce it inside an incomplete application. A new research paper introduces ProgramDistill, a benchmark designed around exactly that scenario.
ProgramDistill is built by factorizing applications into features at different levels of granularity, each tied to replayable behaviors that can be executed through its gold patch. The authors describe an automated pipeline, mine-craft-patch, which discovers 1,975 replay-verified behaviors across 26 applications and constructs 4,063 tasks without human intervention.
In evaluations across nine frontier coding agents, GPT-6 Astra and Claude Opus 5 reached 49.2% and 28.8% success on cumulative workflows in full-application reconstruction. In partial-application reconstruction, success dropped from 100% to 64.0% and from 96% to 32% as restoration depth increased from 1 to 8.
The authors position ProgramDistill as a scalable benchmark with controlled difficulty for evaluating and diagnosing coding agents, and as a natural basis for future curriculum-based training. A public release of the benchmark is said to be in preparation.