LLM-based agents have shown remarkable proficiency in software engineering tasks when an existing codebase provides context, but building a program from scratch remains a formidable challenge. Recent benchmarks like ProgramBench highlight this difficulty: given only natural-language documentation and an execute-only binary as a behavioral oracle, even frontier models solve fewer than 1% of instances.
Existing frameworks often conflate documentation reading, behavioral exploration, and code synthesis into a single pass. This leads to insufficient probing, loss of behavioral intent as context drifts, and propagation of early misinterpretations into the final implementation. Inspired by classical requirements engineering, researchers at Hugging Face propose that behavioral specification elicitation should be a first-class phase preceding implementation.
SpecFirst is a two-stage framework that enforces specification elicitation before code synthesis. A dedicated specification agent first probes the binary and combines observations with documentation to produce a structured specification. Then, a code synthesis agent uses this specification as a stable reference for implementation, decoupling behavioral exploration from code construction.
Evaluated on all 200 ProgramBench instances across four models from two families, SpecFirst consistently outperforms the single-loop baseline. It improves test pass rates by 6.9%–21.3% and binary exploration coverage by 9.4%–18.5%, with all improvements statistically significant. Behavioral analysis reveals that agents with a prior specification begin coding earlier and sustain it longer, rather than repeatedly switching between exploration and implementation.
The results suggest that introducing an explicit requirements-engineering phase is a simple yet effective paradigm for from-scratch program construction, with benefits extending across model families and capability levels.