Hugging Face researchers have introduced RealSWE, a new benchmark designed to evaluate coding agents under realistic user requests. The work addresses a critical gap: while coding agents are typically tested on curated GitHub issues from the SWE-bench family, real-world user prompts are often shorter, more casual, and less structured.
The study analyzed real prompts from SWE-chat and problem statements from SWE-bench Verified and Pro, defining a six-category information taxonomy and four dimensions of linguistic style. The analysis found that 88% of real prompts contain only a problem statement, with or without limited context, compared to just 7% of benchmark problems. Additionally, 87% of real prompts are written casually, whereas 94% of benchmark problems are formal.
To quantify the impact, the team built RealSWE, comprising 381 multi-variant task families derived from SWE-bench Verified and Pro. Each variant shares the same underlying task and gold patch but differs in information composition and linguistic style. Evaluating seven contemporary LLMs, they found that realistic inputs reduce resolution rates by an average of 6.4 percentage points and can even change model rankings.
Controlled analysis revealed that including Desired Behavior and Motivation significantly affects performance, while Environment Information and Reproduction Steps add tokens without measurable benefit. Linguistic style had only small, model-dependent effects. The findings suggest that users and agents should explicitly state desired behavior and motivation, which most real prompts omit, to substantially improve LLM software engineering performance.