Researchers have introduced UltraText Bench, a bilingual benchmark designed to evaluate how well image generation models render dense visual text. Unlike earlier tests focused on short strings, the new benchmark targets scenes that require generators to reproduce long strings across multiple regions with correct placement and legibility.
The benchmark contains 432 prompts spanning 24 real-world scene categories and three difficulty levels, split equally between English and Chinese. Each human-reviewed prompt specifies exact strings for four to twelve text regions, paired with structured references covering content, placement, and visual attributes.
Evaluation uses the Q-Judger vision-language model, which scores each generated image against the full reference along four dimensions: text fidelity, text clarity, spatial quality, and scene quality. Across 24 model configurations, these dimensions exposed different strengths and weaknesses.
For example, Z-Image-Turbo gained 3.81 clarity points over Z-Image-Base while losing 14.76 fidelity points under the reported settings. Performance also varied with workload: Qwen-Image-2512's English composite score fell from 86.50 at difficulty level L1 to 42.86 at L3.
Ten participants took part in a human evaluation of the automatic scores. The authors highlight a key challenge: visually clear text can still fail to preserve the requested content. The benchmark repository is available at github.com/LINs-lab/UltraText_Bench.