Researchers have introduced Inverse Distillation Unlearning (IDU), a framework that addresses two limitations of multi-step matching models such as flow and diffusion models: high inference cost and the risk of reproducing unwanted parts of their training data.
IDU formulates distillation as a min-max objective over a data distribution, representing that distribution as a mixture of the forget-set and the generated distributions. By comparing this mixture with the teacher's training distribution, the method recovers only the retained data at the optimum, according to the paper.
A notable practical constraint is that IDU requires only a pretrained full-data teacher and examples from the forget set. It does not need access to retained training examples, extra feature extractors, or classifiers.
Experiments on MNIST and CIFAR-10 under flow-matching and score-based diffusion settings reportedly show that IDU substantially reduces the generation frequency of forgotten classes while preserving generation quality on retained classes.
The authors describe IDU as the first unified framework for simultaneous unlearning and distillation in unconditional flow-matching and score-based models. An accompanying video provides an overview of the method, training procedure, and experimental results.