Large language models often struggle to answer questions about a fixed set of documents when retrieval is not available at inference time. In a new paper, Hugging Face researchers formalize this as document knowledge internalization and propose a three-stage post-training framework called IAR (Inject, Align, Recover) to convert a corpus into parametric knowledge.
The Inject stage transforms documents into multiple training objectives—continuation, rewrite, and instruction-conditioned reconstruction—to expose the model to dense knowledge. Align then fine-tunes the model with answer-only QA supervision to make the knowledge accessible. Finally, Recover merges the domain-adapted model with the base instruction model to restore general capabilities.
Across two corpora (Common Corpus and CCI) and four model families (Llama, Phi, Qwen, SmolLM), IAR outperforms Vanilla SFT on all four reported metrics in 7 of 8 dataset-model settings. On average, it gains 3.6 percentage points in domain QA accuracy and 12.1 points in general performance across IFEval, MMLU, and MSBench.
The authors emphasize that document exposure, QA accessibility, and capability recovery should be optimized separately rather than treated as a single fine-tuning problem. They invite discussion on retrieval-free knowledge acquisition and the trade-off between domain internalization and general capability retention.