Personalized AI assistants are expected to do more than just execute user requests; they should also assess whether those requests are appropriate given the user's current circumstances. However, most existing research has focused on accurately fulfilling requests, often overlooking the need for assistants to consider context and engage in conflict-based refusal.
To address this gap, researchers at Hugging Face have introduced PACE (Personalized Assistants for Conflict Evaluation), a novel dataset designed to evaluate whether models can identify latent constraints—expressed as egocentric knowledge or events—that render seemingly reasonable user requests inappropriate. The dataset pairs user requests grounded in well-defined personas with egocentric knowledge base (KB) facts, requiring models to integrate contextual evidence to determine if a request conflicts.
The challenge lies in the implicit retrieval setting, which hinders direct association between user requests and conflict-inducing knowledge, making it difficult for existing models to identify relevant user-specific facts. To tackle this, the team also proposes PaceMaker, a multi-agent framework where specialized agents coordinate across query reformulation, multi-hop graph traversal, and conflict-aware filtering to retrieve contextually decisive evidence.
Experiments on PACE evaluate both evidence retrieval quality and conflict decision accuracy, showing that PaceMaker consistently outperforms existing approaches. This work highlights the importance of context-aware reasoning in personalized assistants and opens new avenues for research on conflict-based refusal.