LoRA: low-rank adaptation, and where low rank stops being enough

The problem and what came before

Fine-tuning every parameter of a 175-billion-parameter model produces a 350 GB copy per task and needs enough memory for gradients and optimizer state of every weight (Sec 1). Earlier parameter-efficient methods had costs of their own: adapter layers inserted between existing layers add latency at inference, which matters when batches are small; prefix tuning reserves part of the sequence for learned prefixes, reducing the usable context, and is hard to optimize (Sec 3).

The method

For a pre-trained weight $W_0 \in \mathbb{R}^{d \times k}$, LoRA learns $\Delta W = BA$ with $B \in \mathbb{R}^{d \times r}$, $A \in \mathbb{R}^{r \times k}$ and $r \ll \min(d, k)$, and computes $h = W_0 x + \frac{\alpha}{r} BAx$ (Sec 4.1). $A$ is initialized randomly and $B$ at zero, so training starts from the unmodified model. After training, $BA$ can be merged into $W_0$, so inference runs at the original speed, and adapters can be swapped by subtracting one and adding another. In most experiments LoRA is applied only to the attention query and value projections (Sec 4.2). The adaptation post works the parameter counts and checks merging on a toy layer.

The motivation is a hypothesis: the change fine-tuning needs has low “intrinsic rank”, as earlier work found for over-parameterized models generally (Sec 4.1).

The evidence

GPT-3 175B (Sec 5.5, Table 4), accuracy on WikiSQL and MNLI-matched, ROUGE-1/2/L on SAMSum:

LoRA with 4.7 million trainable parameters is within 0.4 points of full fine-tuning on WikiSQL and above it on the other two. Training memory fell from 1.2 TB to 350 GB and checkpoints from 350 GB to 35 MB (Sec 4.2). Similar parity holds on GLUE with RoBERTa and DeBERTa and on natural-language generation with GPT-2 (Tables 2–3).

Analyses (Sec 7). Given a fixed budget of 18 million parameters, spreading it over the query and value matrices beat putting it all in one (Table 5). A rank as small as 1 was competitive on these tasks when adapting query and value (Table 6). The learned update amplifies directions that were already present in $W_0$ but not emphasized, rather than its top singular directions (Sec 7.3), which the authors offer as an explanation of what adaptation does.

What the evidence supports, confounds and leaves open

Supported. For the tasks tested, adapting attention projections with a small rank matches full fine-tuning at a tiny fraction of trainable parameters, memory and storage, with no inference penalty after merging. This made fine-tuning practical on modest hardware and made per-task adapters cheap to store and serve.

Narrow tasks. WikiSQL, MNLI and SAMSum are well-defined tasks with clear formats, the kind of “behaviour” adaptation where a low-rank change plausibly suffices. They do not test adding substantial knowledge or capability. Biderman et al. compared LoRA and full fine-tuning on programming and mathematics, with about 100,000 instruction pairs or 20 billion tokens of continued pre-training, and found that in standard low-rank settings LoRA substantially underperformed full fine-tuning, while better preserving performance outside the target domain; full fine-tuning learned perturbations of 10 to 100 times higher rank than typical LoRA configurations (Biderman et al., 2024, abstract). Low rank is a good fit when the change is small, and a constraint when it is not.

Tuning effort. Each method in Table 4 is reported at chosen hyperparameters from single configurations; differences of under a point between methods should not be over-read.

Left open. Which matrices to adapt beyond attention (later practice often adapts all linear layers), how rank should scale with model and data size, and how LoRA interacts with quantized base models, which QLoRA later addressed.

How it changes decisions

• Use LoRA for behaviour and format adaptation on narrow tasks, and to serve many tasks from one base model.
• Expect a gap to full fine-tuning when the target domain requires substantial new knowledge or skill; try higher ranks or full fine-tuning and compare on a held-out set, including off-target tasks to measure forgetting.
• Merge adapters for latency-sensitive serving; keep them separate when swapping per task or customer.