Text-to-image (T2I) models have made significant strides in generating visually compelling images, yet they often struggle with open-world tasks that demand complex semantic understanding, multi-step reasoning, and the integration of external world knowledge. Existing attempts to inject agentic capabilities into image generation either rely on fixed workflows or only partially automate the process, leaving reasoning, tool invocation, and image generation uncoordinated.
To address this, researchers at Hugging Face propose ToolArtist, a fully agentic image generation model built by post-training a Unified Multimodal Model (UMM). ToolArtist dynamically orchestrates reasoning, external tool use, and native image generation within a single unified policy, enabling more flexible and adaptive behavior.
The training process involves two key stages. In Supervised Fine-Tuning (SFT), a teacher agent is equipped with search tools and an image-generation tool, and the resulting trajectories are converted into a UMM-compatible format where the image-generation tool is concealed but the generated images are retained. In Reinforcement Learning (RL), the team develops an agentic RL infrastructure for UMMs and introduces Reason-Act-Draw GRPO (RAD-GRPO), which uses complementary intent and quality rewards to jointly optimize the model.
Experiments demonstrate that placing the entire open-world image-generation process under an agent policy consistently outperforms approaches with fixed pipelines or only partially agent-controlled components. The researchers release the training data and the complete post-training infrastructure to facilitate further research.