Hugging Face has published a new research paper introducing VākQA, described as the first benchmark for Telugu spoken factoid question answering. The work addresses a gap in spoken question answering (SQA) resources, which have largely focused on high-resource languages.
The benchmark comprises 2,001 factoid question-answer pairs spanning six domains, along with 2.53 hours of speech audio, bilingual transcriptions, and human-verified reference answers. According to the authors, the reliability of automatic evaluation in this setting had not previously been quantified.
As part of the study, the researchers validated evaluation methods against human judgments. They report that Gemini-as-a-judge best approximates human ratings, though it is non-uniformly strict, while open-weight judges systematically penalize correct Telugu answers that differ in surface form from the reference.
Using this validated setup, the team benchmarked proprietary and open-weight models across input modality, language, and domain. The paper notes that Telugu phrasing retains cultural specificity that is lost in translation, speech input introduces phonetic confusions that can alter question meaning, and cascaded ASR-MT errors compound progressively.
VākQA is publicly released, and the paper also provides baseline results for proprietary and open-weight models under varying input conditions.