Industrial recommender systems increasingly rely on a pretrain-then-transfer paradigm, but behavioral distribution drift poses two key questions: what to learn from user behavior sequences, and how to transfer that knowledge when the pretrained model is continually refreshed. A new paper from Hugging Face, titled "Knowledge-Geometry Decoupling: Refreshable Pretrained Transfer for Streaming Recommendation," proposes a solution called Knowledge-Geometry Decoupling (KGD).
For the first question, the authors argue that conventional next-token prediction treats adjacency as dependency, potentially encoding spurious transitions across unrelated sessions. To address this, they introduce Behavioral Multi-Token Prediction (BMTP), which retains only collaboratively or semantically related future items as supervision, yielding cleaner and more transferable behavioral knowledge.
For the second question, KGD separates pretrained knowledge and task-specific geometry into distinct parameter sets. A refreshable encoder owns behavioral knowledge, while a task learner reads contextualized encoder states through read-only cross-attention and writes task-specific geometry via Anchored Calibration Residual (ACR), which is orthogonal to the pretrained embedding. This decoupled ownership enables continual knowledge refresh without task-gradient interference or invalidating downstream adaptation.
In experiments, KGD improves over strong pretrain-transfer baselines by 4-12% on eight public benchmarks and sustains its advantage over a 90-day production stream where baselines show no gains. The method has been fully deployed at Shopee, where a live A/B test on Homepage Search showed a 1.75% increase in GMV per user and a 1.53% increase in advertising revenue. The core implementation is available on GitHub.