Low-Rank Adaptation (LoRA) has become a cornerstone of parameter-efficient fine-tuning, but its training dynamics are not always stable. A new paper from Hugging Face researchers introduces Normalized Low-Rank Adaptation (NoRA), a method that normalizes the down-projection matrices during training to stabilize optimization and boost performance.
The key insight is that because LoRA initializes the up-projection to zero, early training is governed almost entirely by the down-projection. By normalizing these matrices, NoRA accelerates convergence, improves performance, and enhances training stability across pretraining, supervised fine-tuning, and reinforcement learning. It also helps mitigate catastrophic forgetting.
Remarkably, NoRA can be applied only at initialization to yield similar benefits, meaning standard LoRA can be improved without repeated normalization throughout training. The method requires no additional trainable parameters and adds no inference-time computation, making it a broadly applicable enhancement to existing LoRA workflows.