Enterprises deploying retrieval-augmented generation (RAG) routinely face a difficult operational question: should a given system version be promoted, revised, or blocked? The evidence behind that call is often incomplete, and the metrics come from fallible LLM judges. A new paper accepted at NFMCP 2026 (ECML PKDD 2026 Workshops) introduces AGO AI Quality Gate, an evidence-first framework built and deployed at Protom for industrial RAG assessment engagements.
AGO combines four components: a four-state decision model that treats missing data and judge errors as explicit outcomes rather than silent passes; layered scoring that mixes deterministic checks, local guardrails, and structured LLM evaluation; a stratified beta-binomial gate that quantifies regression risk probabilistically; and a mandatory meta-evaluation protocol that validates the LLM judge before it is allowed to influence decisions.
Because engagement data is proprietary, the authors evaluated the judge layer on RAGBench, a public benchmark of 100,000 annotated RAG traces spanning 12 datasets. On identical stratified test samples of 1,200 per judge, a low-cost judge (gpt-4.1-nano) detected non-adherent answers barely above chance, with an AUROC of 0.603 [0.570, 0.634], despite producing flawless protocol output. The stronger gpt-4o reached 0.783 [0.756, 0.807], yet its per-domain performance still ranged from 0.62 to 0.88.
A fixed-seed gate study covering regression, no change, and improvement scenarios quantified unsafe promotion, false-alarm cost, and improvement throughput. Under regression, the decision-grade profile reduced unsafe promotion to 22.2%–35.1%, compared with 29.3%–41.8% for a naive gate.
The authors conclude that judge quality must be measured per engagement rather than certified once, and that point estimates alone are not a release decision. Missing evidence, they argue, should be a first-class outcome — never a silent pass.