Large language models (LLMs) are increasingly used as mobile assistants, but their ability to leverage personal information scattered across multiple apps remains poorly understood. To address this, researchers have introduced SPIEval, a human-curated benchmark designed to evaluate LLMs on tasks that require reasoning, disambiguation, integration, preference inference, and multi-intent decomposition.
SPIEval comprises 250 tasks spanning 4,335 personal records distributed across 10 apps, with multi-turn interaction supported through 21 tools. The benchmark is designed to feature diverse scenarios, challenging tasks, scattered information, controllable environments, and verifiable outcomes.
Evaluation of nine representative LLMs reveals substantial room for improvement. The best-performing model, GPT-5.5 (xhigh), achieved only 57.3% accuracy, while the weakest scored just 16.4%. Further analysis shows that 79% of failures stem from inaccurate information localization, as models often commit to plausible but incorrect information instead of continuing retrieval for verification.
Additionally, fewer than 2% of retrieval actions employ advanced search methods, and there is substantial variation in search efficiency across models. These findings expose fundamental limitations of current LLM-based mobile assistants and motivate future research. Data and code are available at https://huggingface.co/datasets/Junjie-Ye/SPIEval.