Researchers have released CADWorld, a new benchmark designed to evaluate computer-use agents on long-horizon tasks inside FreeCAD, a professional mechanical computer-aided design environment. The work targets a gap the authors identify in existing desktop-agent evaluations: most benchmarks do not cover engineering workflows whose outputs are persistent, structured artifacts.
According to the paper, mechanical CAD is especially demanding because an agent must manipulate geometry and constraints across long interaction sequences while producing a native project whose dimensions, construction structure, and downstream engineering state remain valid. CADWorld contains 200 tasks spanning 11 mechanical-CAD workflow categories, including sketching, part modeling, assembly, CAM, FEM, measurement, mesh processing, and technical drawing.
Agents interact with the software through screenshots and GUI actions. Success is not judged by visual similarity but by task-specific executable checks run over saved FreeCAD artifacts and auxiliary outputs, covering geometric properties, parametric structure, constraints, manufacturing state, and simulation results.
Across seven current agents evaluated on the full benchmark, the strongest achieved 17.5% success, compared with an 87.0% expert reference pass. The authors report a qualitative split in failure modes: weaker agents often fail before producing a valid artifact at all, while stronger agents increasingly fail on structural, geometric, and construction-process requirements.
The paper concludes that CADWorld exposes a gap between general GUI competence and reliable execution of persistent, verifiable engineering workflows. The project is accessible at cad-world.github.io.