Evaluating how large language model assistants handle everyday social advice has long been difficult: the assistant must infer a situation from a user's subjective narrative, and properties such as another person's intentions rarely have a verifiable ground truth. A new paper, Verifiable Social Reasoning for LLM Assistants, proposes a framework called Fuse to address both problems.
Fuse is a multi-agent simulation framework for studying user-mediated social reasoning. In the setup, a target agent with a hidden motive interacts with other agents, including one representing the user. That user then consults the assistant under evaluation, asking it to infer the target's motive. Because the motive is defined by construction, the framework supplies verifiable ground truth for the assistant's answer.
The authors report validating the faithfulness of the simulation through a human study involving 24,000 annotations. They then applied Fuse to 12 LLMs, using it to isolate several factors that shape performance.
According to the paper, the results show that user mediation compounds the inherent difficulty of social reasoning, that LLMs are systematically sensitive to biased user framing, that models can require more details than humans need to reach a correct prediction, and that longer conversations do not always improve performance even when they offer opportunities for clarifying questions.
The researchers have open-sourced Fuse along with a dataset containing 21,000 examples. The paper is available on Hugging Face.