Large language models (LLMs) are increasingly used to turn natural-language descriptions into optimization models, but real-world operations research (OR) requests are often incomplete. Missing objectives, constraints, or business rules can drastically change the resulting mathematical program. Existing evaluations usually assume a complete specification, overlooking whether an agent knows when clarification is needed before modeling.
To address this, researchers from Hugging Face introduce OR-Clarify, a benchmark for pre-formulation clarification. Each task presents a partial public problem description, withholds structured hidden slots, and evaluates agents through bounded interaction with a simulated user. The benchmark supports both open-ended and choice-based clarification, measuring slot recovery, stopping behavior, silent assumptions, and interaction cost.
The team also proposes Interactive Optimization (InterOPT), a two-stage framework that identifies unresolved formulation-critical gaps and uses them to decide whether to ask the next question or stop. In choice-based experiments, InterOPT substantially outperforms all baselines in exact slot recovery; in the open-ended setting, it remains competitive with strong prior methods.
Together, OR-Clarify and InterOPT reframe OR assistance as a selective completeness decision: clarify when needed, stop when ready, and quantify what remains missing. The project is open-source, with implementations and datasets available on GitHub and Hugging Face.