Today’s leading language models contain upwards of a trillion parameters, pretrained on tens of trillions of tokens. Base model performance keeps improving with scale, as these trillions are necessary to learn the vast amount of knowledge and capabilities shown by these models.

In contrast, post-training involves smaller datasets and generally focuses on narrower domains of knowledge and ranges of behavior. It seems wasteful to use a terabit of weights to represent updates that could be compressed into a much smaller space. The leading PEFT method is low-rank adaptation, or LoRA. LoRA replaces each weight matrix W from the original model with a modified version W′ = W + γBA, where B and A are matrices with far fewer parameters than W, and γ is a scaling factor.

LoRA may offer advantages in the cost and speed of post-training, and there are also a few operational reasons to prefer it to full fine-tuning: multi-tenant serving, since LoRA trains an adapter while keeping the original weights unchanged, a single inference server can keep many adapters in memory and sample from them simultaneously in a batched way.

These reasons are sufficient to explain the growing popularity of LoRA since the publication of the original LoRA paper in 2021. However, there is agreement that LoRA underperforms in settings that resemble pre-training, namely those with very large datasets that exceed the storage capacity of the low-rank matrices.

In our experiments, we find that indeed, when we get a few key details right, LoRA learns with the same sample efficiency as full fine-tuning and achieves the same ultimate performance. This article covers a series of supervised fine-tuning and reinforcement learning experiments we conducted to determine the conditions under which LoRA matches full fine-tuning efficiency.