As large language models increasingly serve as automated evaluators, the cost and reliability of those judgments become pressing concerns at scale. A newly posted paper, JEV-as-a-Judge: Accept When Confident, Escalate When Unsure, examines whether a lightweight, decision-only judge can handle a first pass of evaluation and flag the cases that need closer scrutiny.
The authors compared their approach against sixteen generative and reward-model judges, using blinded human adjudication as the reference. On ordinary preference judgments and evidence-grounded factuality, the method came within three percentage points of the strongest comparator — described as a state-of-the-art LLM judge — while costing just 0.36% of that comparator's fee.
The picture changes on harder tasks. Larger gaps emerged when a judgment required checking a derivation or resisting an elaborately written but incorrect answer. On several benchmarks, the shortfall relative to the strongest comparator was concentrated in low-confidence decisions.
That pattern points to a practical fix: a frozen cascade that accepts confident verdicts and escalates uncertain ones. According to the paper, this setup retains 99% of the comparator's accuracy at a lower cost, suggesting that selective escalation can balance economy and reliability in large-scale evaluation.