Researchers have released RealCompanion, a benchmark designed to test whether an AI companion that converses with a person over months actually comes to understand them — remembering what was said, inferring who the person is, and recognizing when the past is relevant to the current message.
The authors argue that testing this capability requires a real person's record, but such records are private. Existing benchmarks therefore generate the person and the questions and decide in advance what matters. RealCompanion takes a different route: it is built from ten real relationships with an AI companion, comprising 27,218 messages over up to 120 days.
The dataset is released as the conversation itself plus four derived files: a profile, a persona, a chat ground truth, and a question set — each citing the messages it rests on. Every chat label carries the reasoning trace that produced it, checked stage by stage against the conversation.
Three findings stand out. First, the past is rarely needed and often far away: pooled measures mislead, since a recency window finds the required message for 95.9% of probes but only 2.2% of those that need memory, and at the natural rate 96% of the gain from supplying recorded evidence comes from messages that need none. Second, no detector the authors tried could tell when memory is needed on real messages; authored questions over the same histories leak the cue, and labeling the same messages as memories raises their use by ten to fourteen points. Third, three agent systems reconstruct the persona with the same F1 score at a 31-fold difference in cost.
The paper is available on Hugging Face, and the authors invite collaboration by email.