Fréchet distance has recently become a popular distribution-level objective for fine-tuning generative models after initial training, complementing traditional sample-level losses like diffusion and flow matching. However, directly optimizing Fréchet objectives can lead to a phenomenon known as 'Fréchet hacking,' where the target metric improves but actual visual quality and alignment in other feature spaces stagnate or decline.
Researchers attribute this failure to the static, pretrained feature spaces used by existing Fréchet losses. These fixed representations provide incomplete views of the differences between real and generated data distributions, limiting the optimization's effectiveness.
To address this, the team introduces Adversarial Fréchet Distance (AdvFD). AdvFD augments the static Fréchet objective with a learnable representation that adversarially maximizes the Fréchet discrepancy between real and generated samples, while the generator simultaneously minimizes this discrepancy in the resulting adaptive feature space. This dynamic interplay helps the model capture more relevant distributional differences.
A key innovation is real-feature whitening, which normalizes the scale and covariance geometry of the adversarial representation. This prevents the representation from trivially inflating the objective through feature amplification, stabilizing the min-max optimization process.
Extensive experiments demonstrate that AdvFD consistently improves one-step generator post-training across both JiT and pMF backbones and across different model scales, suggesting broad applicability for enhancing generative model quality.