Enterprise workflows increasingly depend on AI agents to perform schema-guided extraction: given a document and a user-defined schema, the agent must faithfully follow the schema and produce the correct output, with source evidence as grounding metadata. To evaluate these systems, researchers at Hugging Face have introduced ExtractBench, a new benchmark that, for the first time, scores value accuracy, record completeness at scale, grounding, and measured cost together.
The benchmark comprises 4,869 pages across 370 enterprise documents, spanning 8 business domains and 67 document types, with clear tags distinguishing different challenge scenarios. The authors developed a scalable curation pipeline that combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms, ensuring high-quality ground truth.
Evaluation uses order-insensitive value F1 for accuracy, plus two grounding metrics for source traceability: word-level and page-level F1. The results reveal notable trade-offs: commercial vision-language models (VLMs) perform well on short documents but often truncate record lists on long ones, while coding agents maintain higher accuracy at a significantly higher cost.
Notably, LlamaExtract Agentic Plus ranks first on all three metrics, achieving accuracy comparable to coding agents at a fraction of the cost. The dataset and evaluation code are publicly available on Hugging Face and GitHub, inviting further research and benchmarking in this critical area.