Hugging Face has published a research paper titled FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation. The work addresses a common split in multimodal AI, where a contrastive or self-supervised visual encoder is trained first and a separate generative model is trained afterwards, leaving generation constrained by frozen embeddings.
FLAT proposes a single representation pre-training framework that jointly optimizes a shared multimodal encoder alongside downstream text-to-image (T2I) and image-to-text (I2T) decoders. By combining contrastive alignment with bidirectional cross-modal generative objectives, the authors aim for representations that serve both as discriminative semantic descriptors and as generative conditions.
Architecturally, FLAT maps visual and textual inputs into a unified continuous 1D sequence space and applies nested dropout over prefix-K tokens to allow dynamic output lengths. According to the paper, a single pre-training stage enables cross-modal retrieval and generation across variable prefix K, reaching a T2I GenEval score of 71.1.
With task-specific fine-tuning, the reported results include 83.1 GenEval for T2I generation; 40.5 BLEU-4 and 138.6 CIDEr on MS-COCO image captioning; and Recall@5 scores of 86.8 (I2T) and 75.8 (T2I) on MS-COCO, alongside 98.3 (I2T) and 93.6 (T2I) on Flickr30K. The authors also describe qualitative evaluations showing that FLAT representations natively support linear interpolation, latent space arithmetic, and zero-shot composed retrieval.