Image tokenizers define the "visual language" of unified multimodal models, yet they are typically evaluated through isolated metrics or generation- or understanding-only benchmarks. A new research paper argues these approaches fail to capture how visual tokens behave when modeled jointly with text, and proposes a different lens: treating image tokenizers as visual languages.
The authors build a controlled pure-autoregressive testbed and track task-specific validation losses during multimodal continual pretraining across four prediction tasks: text, image, text-to-image (T2I), and image-to-text (I2T). They examine how these losses scale and relate to downstream performance, using them to study multimodal learnability and tokenizer design.
The study reports four main findings. First, losses should be analyzed by task, since they exhibit distinct scaling behavior and rank tokenizers differently. Second, the loss-performance relationship depends on the predicted token space: for a fixed tokenizer, T2I and I2T losses correlate with generation quality, but across tokenizers the T2I loss-performance relationship shifts with the image-token space, whereas I2T loss, computed over a shared text vocabulary, provides a more consistent signal and also correlates with both generation and visual understanding performance after supervised finetuning.
Third, better reconstruction does not necessarily yield lower task-specific losses or stronger downstream performance. Fourth, image tokenizer choice can affect text modeling under joint optimization. As case studies, the authors revisit three tokenizer design axes — the discriminator, semantic supervision, and vocabulary size — to examine their effects on joint modeling and downstream performance.
The paper concludes that its testbed offers a complementary perspective on image tokenizers as visual languages, highlighting their interplay with text in joint multimodal training. The authors argue that image tokenizers should be designed and evaluated as visual languages, not just as image compressors. Code and a project website accompany the paper.