Hugging Face researchers have introduced PALATE (Person-Aligned LLM-Simulated-User Assessment with Tailored Evaluation), a scalable benchmark for evaluating role-playing agents (RPAs) in interactive, multi-turn settings. The work addresses two key limitations in existing benchmarks: their reliance on fixed dialogue histories and the use of generic rubrics that may not reflect individual user satisfaction.
The benchmark includes a pool of 300 character profiles and trains five per-user simulators that engage candidate RPAs in free-form conversations. Alongside a general quality rubric, PALATE constructs personalized rubrics to measure user satisfaction, which show higher agreement with human judgments on held-out data.
In an evaluation of 16 candidate RPAs, PALATE separately characterizes generic turn quality, long-horizon session capability, and per-user experience, producing interpretable evaluations of specific user-RPA pairs rather than a single user-independent ranking.
The paper also lists related works recommended by the Semantic Scholar API, including benchmarks for human-centered dialogue and emotion management, highlighting the growing focus on user-centric evaluation in AI.