Researchers at Hugging Face have introduced AdaPop (Adaptive Popularity), a novel method for machine unlearning in large language models (LLMs). The approach addresses a key limitation of existing techniques: they apply uniform gradient pressure regardless of how frequently a fact appeared in training data. However, popular facts are memorized more deeply and resist removal longer than rare ones.
AdaPop combines local token confidence with a per-fact popularity-dependent exponent, derived from external proxies such as Wikidata sitelinks or an LLM-as-Judge. It also automates the delicate balance between forgetting and retaining knowledge using a dual-ascent controller that adjusts the retain penalty each epoch.
In evaluations across three model families and two benchmarks, AdaPop leaked approximately 5 times less forgotten content than competing methods under paraphrased queries, and about 1.6 times less under adversarial reformulations. Internal metrics further showed that forget-set hidden states moved further away from the pre-unlearning model's states, while retain-set representations remained close.
This work highlights the importance of considering data frequency in unlearning, offering a more effective approach to privacy and compliance in AI systems.