A new research paper from Hugging Face proposes a method called program distillation to address the high cost, latency, and opacity of LLM-as-a-judge evaluation. Instead of prompting a large language model each time, the approach distills the judge's decision logic into a committee of programs that score outputs directly.
The system, named PAJAMA, synthesizes programs as judges, aggregates their decisions into a joint verdict, and includes a fallback mechanism that escalates low-confidence cases to an LLM. This hybrid design balances efficiency and accuracy.
Experiments across five datasets and four model families show that programmatic judges can match the performance of a 13B-size LLM judge. When used as routing signals, PAJAMA improves both accuracy and throughput, advancing the Pareto frontier.
Beyond evaluation, the distilled programs produce cheap reward signals. On RewardBench, a reward model trained on program verdicts outperforms one trained on proprietary LLM labels at two orders of magnitude lower API cost.
The authors highlight that programmatic judges are transparent, easily inspected, and editable—turning evaluation criteria into version-controlled code. This could significantly reduce API bills for stable evaluation tasks.