Researchers have released Duplex-MPE, a new benchmark designed to evaluate how full-duplex speech assistants behave when they are not the sole conversation partner. Unlike prior benchmarks that focus on a designated user, Duplex-MPE places the assistant inside a shared conversation among several people, testing whether it should answer, remain silent, or stop speaking.
The benchmark contains 2,000 scenarios, each involving three or four human speakers plus one assistant. Every request appears in two paired forms — one where the assistant is addressed explicitly and one where it is addressed implicitly — allowing researchers to measure how addressing style affects model behavior. Models receive continuous conversation audio without transcripts or supplied turn boundaries.
Four separate scores track fresh response initiation, answer accuracy, silence preservation, and stopping when a human resolves a request. Five open-weight speech systems were evaluated: MiniCPM-o 4.5, Moshi, FLM-Audio, Voila, and Freeze-Omni. According to the paper, MiniCPM-o 4.5 leads on three of the scored capabilities, while other systems often speak frequently yet produce inaccurate answers or fail to remain silent when appropriate.
A transcript-based Gemini 3.1 Pro reference responded 64.3 percentage points more often to explicit than implicit requests. Paired tests found no significant response-rate difference between explicit and implicit addressing for the speech systems evaluated.
The team says audio examples and benchmark design details are available online, with data and code planned for release before December.