Speech brain-computer interfaces (speech BCIs) hold promise for restoring communication in individuals with paralysis and enabling novel human-computer interaction. However, the field has struggled with a lack of standardized evaluation, as systems vary in datasets, recording methods, speech types, and vocabularies, making reported results incomparable.
To address this, researchers from Hugging Face and collaborators propose open-vocabulary mutual information (OVMI), an information-theoretic measure that quantifies the information conveyed by a decoder relative to a reference distribution over words a user might wish to communicate. This allows systems tested under different conditions to be compared on a common communication scale.
The team demonstrates that traditional metrics like word error rate (WER) and accuracy, computed only over supported words, can overstate a system's actual communicative capability. Using OVMI, they expose trade-offs between vocabulary coverage and decoding accuracy, showing that these comparisons depend on the expected user communication.
Furthermore, selecting a vocabulary to maximize OVMI yields up to 16.3% relative improvement in accuracy across three speech domains. The authors argue that OVMI provides a principled framework for comparing heterogeneous systems, improving vocabulary design, and tracking progress in the field.