In a new paper, researchers from EPFL and Hugging Face present Modus, a decoder-only any-to-any multimodal model that can predict any modality from any combination of others within a single network. Unlike existing any-to-any models that rely on encoder-decoder or diffusion architectures and are trained from scratch, Modus leverages strong pre-trained decoder-only models as a prior, improving performance and flexibility.
The key innovation is treating all modalities symmetrically: every modality serves as both input and output, without modality-specific heads, losses, or task pipelines. This unified design enables a range of applications, including chained generation through intermediate modalities and cross-modal self-verification, where the model scores its own outputs using another generated modality.
Modus demonstrates strong out-of-the-box performance and is competitive with specialist and multitask baselines across various benchmarks, all with a single model. The authors highlight that the model uses one decoder, two experts, and zero task heads, simplifying the architecture while maintaining high quality.
All materials, including code and models, are open-sourced at modus-multimodal.epfl.ch.