Researchers have introduced IndicBankBench, a benchmark of 799 synthetic cases designed to measure how safely and reliably language model assistants operate in Indian retail banking. The work targets a gap the authors argue is easy to miss: judging an assistant only by its final answer overlooks errors that occur along the way.
As the paper notes, a banking assistant may ask for information it already holds, rely on stale context, select the wrong account, or state a correct value and then write an invalid one. To capture such failures, cases span five operational domains, a capability/refusal domain, and twenty primary axes, and are evaluated in a mock banking environment.
Evaluation proceeds through four ordered stages: safety, action and tool use, response adequacy, and advisory quality. Tool use and most safety checks are deterministic, a narrow resolver handles only ambiguous confirmation-before-write cases, and a separate LLM judge assesses semantic response adequacy. This staged design lets the harness pinpoint where a case failed rather than only whether it did.
Each case is run three times, and the benchmark reports strict pass^3, which requires success on all trials. Across the eleven evaluated models, strict reliability ranged from 43.7% to 58.2%, while at-least-once success ranged from 60% to 74%. The authors say this gap shows that at-least-once success can overstate dependable banking behavior.
Case-level diagnostics further distinguish systems that ask unnecessary questions from those that act but fail to reconcile customer context or fully resolve the request. The cases, mock environment, and evaluation harness have been released, and the team says it welcomes feedback on case design and grading rules.