Visual generation is increasingly demanding high-resolution images, long videos, and multimodal context, making the quadratic cost of full attention a major bottleneck. To address this, researchers at Hugging Face have introduced Chimera, a hybrid visual diffusion backbone designed with a principled scaling recipe.
Chimera processes text, image, and video tokens in a single raster-ordered stream without positional embeddings. It combines Kimi Delta Attention (KDA) for long-context state tracking with O(N) complexity, interleaved Multi-head Latent Attention (MLA) for direct global interaction, and modality-aware short convolutions for local spatiotemporal context. Sparse Mixture-of-Experts (MoE) layers expand capacity while keeping activated compute in check.
To scale this heterogeneous architecture, the team developed HeteroP, a module-wise scheme that transfers hyperparameters across width and depth based on each tensor's functional fan-in and model depth. This yields a consistently tuned family of models used to fit Chinchilla-style compute-optimal laws for activated model size, training-token count, and image-video data ratio.
Guided by these laws, they trained an 11B-parameter Chimera with 2B activated parameters. Results show three key findings: (1) the dense backbone is 1.7x as compute-efficient as a matched full-attention Wan-2.1 2B baseline, while the complete system reaches 7.3x; (2) without length-specific fine-tuning, Chimera extrapolates zero-shot from 5-second training clips to 30-second videos, with only 6.5% FID degradation in the last five seconds; (3) the fitted laws indicate that compute-optimal image pretraining divides compute nearly evenly between activated model size and training-token count, whereas video pretraining modestly favors model size at higher budgets.
These results establish a foundation for designing and scaling efficient long-context diffusion architectures, potentially impacting future video generation and multimodal AI systems.