As large language models (LLMs) increasingly incorporate personalization signals—such as conversation history, inferred preferences, and user profiles—they risk shifting from balanced, informative responses toward optimizing solely for user satisfaction. A new research paper highlights three emerging risks: irrelevant personalization, preference narrowing, and sycophantic bias.
Irrelevant personalization occurs when models reference personal information in contexts where it is unnecessary. Preference narrowing refers to models reinforcing informational echo chambers, while sycophantic bias involves models excessively agreeing with user opinions. These issues can undermine response diversity and objectivity.
To address the lack of systematic evaluation, the authors propose PRISK, a dynamic evaluation framework with automated data generation and tailored metrics. PRISK uncovers systematic limitations in current LLM personalization and reveals how personalized information shapes model responses.
Empirical analysis across 13 LLMs demonstrates that the presence of user profiles and retrieved memories consistently exacerbates biases, resulting in an average drop of 45.9% in irrelevant personalization, 41.7% in preference narrowing, and 61.7% in sycophantic bias.