Representation alignment (REPA) has become a common technique for accelerating diffusion transformer training, but its alignment targets have been dominated by semantic encoders such as DINOv2 and CLIP. A new paper, PixelDense, argues that this focus overlooks dense-prediction foundation models that encode spatial structure — the very signal recent analysis identifies as the carrier of the alignment effect.
The authors first test dense-prediction teachers in pixel-space diffusion. SAM2, Depth Anything v2, and Metric3D v2 each outperform a DINOv2-only GenEval baseline, with the two geometric teachers leading the segmentation teacher. However, a flat sum of all four teachers performs worse than the best single geometric teacher, which the paper attributes to semantic and geometric gradients competing for a single denoiser projection.
PixelDense addresses this by routing DINOv2 and SAM2 through a semantic projection stream and Depth Anything v2 and Metric3D v2 through a geometric projection stream, adding a weight-space orthogonality penalty that keeps the two streams in disjoint subspaces. All four teachers remain frozen during training and are dropped at inference.
Applied to PixelGen and DeCo with a single recipe, PixelDense improves GenEval, DPG-Bench, and HPS v2.1, and raises PixelGen-XXL's GenEval Overall from 0.7927 to 0.8093, beating every single-teacher and unfactored multi-teacher variant. In partial-noise reconstruction, independent panoptic, depth, and surface-normal probes show up to 53.1% PQ gain and 36.0% depth AbsRel reduction at τ=0.5 across COCO and Flickr30K. From random initialization, PixelDense reaches the baseline's peak GenEval 1.23x faster, and in SDEdit editing on PIE-Bench it preserves more source background and layout at every edit strength, raising background PSNR by up to 2.2 dB.