A newly posted research paper on Hugging Face examines whether a plain Transformer can generate high-quality images directly in pixel space, without a variational autoencoder (VAE) or a specialized pixel decoder. The authors argue that it can, provided the model's internal representation is both properly formed and actively exploited.
The work reports two central findings. First, better representation quality does not necessarily translate into better generation: externally aligned semantics can be progressively lost during pixel synthesis, and REPA-style alignment improves semantics at the aligned layer but the gain fades in later layers and does not improve final FID. Second, architectural separation with specialized pixel decoders mainly redistributes semantic and spatial modeling rather than producing representations better suited to generation.
To address this, the authors propose Representation Guidance (RG). A low-dimensional bottleneck compresses generation-relevant information into a compact implicit latent, which is then used to guide decoder generation by modifying the flow-matching target according to its induced prediction offset, expressed as v_RG = v + (s-1)·sg(Δv_rep). The authors describe this as functioning like self-distillation in self-supervised learning, where the representation is bootstrapped from the model's own predictions.
On ImageNet 256x256, RG-L/16 with 459M parameters is reported to outperform substantially larger pixel diffusion transformers, while RG-XL/16 (725M) reaches an FID of 1.55. The method also reports 4× faster convergence, with RG-B reaching JiT-B's 600-epoch FID in 150 epochs. At 512x512, RG reaches an FID of 1.74, and it extends to text-to-image generation with 0.89 on GenEval and 87.1 on DPG-Bench.
The authors state that RG scales consistently from B to XL, achieving lower FID with fewer parameters and FLOPs than plain-Transformer JiT baselines while also outperforming pixel-specialized decoders at lower complexity. Project and code links are provided by the authors.