A new research paper from Hugging Face, titled Scaling Laws for Hypernetwork-Based Knowledge Injection in Large Language Models, explores the use of hypernetworks for injecting factual knowledge into large language models (LLMs) at scale. The authors propose a novel approach where a hypernetwork is trained to generate a fixed LoRA adapter that, when inserted into the target model, enables it to answer questions about a large corpus of facts.
The study introduces a large-scale dataset called MegaWikiQA, containing tens of millions of multi-hop question-answer examples across 39 domains, constructed from Wikidata5M. This dataset allows for rigorous investigation of scaling behavior along hypernetwork depth, width, and target network size.
Key findings include: (i) hypernetwork-based injection exhibits broadly predictive power law scaling along all architecture axes, and (ii) hypernetworks achieve reliable out-of-distribution (OOD) generalization at increasing scales, with steeper scaling exponents compared to LoRA fine-tuning and full fine-tuning in all OOD evaluations.
The authors conclude that hypernetworks provide a principled and scalable substrate for train-time adaptation, offering a promising alternative to existing methods. This work establishes the first empirically grounded scaling laws for hypernetworks in factual reasoning tasks.