LoRA & Quantization

Two tricks that let you fine-tune and run billion-parameter models on a single consumer GPU. LoRA shrinks what you train; quantization shrinks how big each weight is. Together (QLoRA) they're why open-model fine-tuning is within reach of a laptop budget.

LoRA / low-rankparameter-efficient fine-tuning quantizationINT8 / INT4QLoRA

The problem: big models are expensive to adapt and to serve

A 7-billion-parameter model in full FP32 precision is ~28 GB just to store the weights — and fully fine-tuning it is far worse: you also need memory for gradients and optimizer state (Adam keeps two extra numbers per weight), easily pushing past 80–100 GB. That's data-center territory for what might be a small customization. LoRA and quantization attack the two costs separately.

LoRA — train a tiny low-rank "patch" instead of the whole matrix

Fine-tuning asks: find an update ΔW to add to a weight matrix W so the model does your new task. The key empirical insight behind LoRA (Low-Rank Adaptation) is that this update has low intrinsic rank — it doesn't need to be a full, dense d × d matrix. So instead of learning ΔW directly, LoRA freezes W and learns ΔW = B · A, where A is r × d and B is d × r with a tiny rank r (often 8 or 16). The forward pass becomes y = Wx + (α/r)·BAx — the frozen model plus a small learned correction.

The payoff is dramatic: you only train the 2·r·d numbers in A and B instead of d². Pick a dimension and rank and watch the trainable fraction collapse:

weight dimension d 4096

LoRA rank r 8

Three more things make LoRA a favourite: the adapters are tiny files (megabytes, not gigabytes), so you can keep dozens and hot-swap a different "personality" per request; you can merge BA back into W after training so there's zero extra inference latency; and because the base is frozen, you mostly avoid catastrophic forgetting. (Initialised so B = 0, the model starts exactly as the original and only departs as it learns.)

Quantization — store each weight in fewer bits

The other cost is sheer size, and most of a weight's 32 bits are wasted precision. Quantization stores weights using fewer bits — FP16 (2 bytes), INT8 (1 byte), even INT4 (½ byte) — by mapping the range of real values onto a small grid of representable levels and rounding each weight to the nearest one. Fewer bits = fewer levels = coarser rounding, but neural nets are remarkably robust to this small added noise. Pick a precision and watch both the model's memory footprint and the rounding error change:

precision

a sample weight value 0.37

QLoRA — the combination that put fine-tuning on one GPU

QLoRA stacks both ideas: load the big base model quantized to 4-bit and keep it frozen, then train small LoRA adapters in higher precision on top. The frozen 4-bit base slashes the memory needed to hold the model, while LoRA slashes the memory needed to train it (few trainable params, so little gradient/optimizer state). The famous result: fine-tuning a 65B model on a single 48 GB GPU, or a 7B model on a 12–16 GB consumer card — something that used to require a cluster.

Tradeoffs & when to use them

LoRA is the default for adapting an open model to a domain or style on a budget. Full fine-tuning still wins at the very frontier of quality, but rarely by enough to justify the cost.
Higher rank r = more capacity to learn but more params; r = 8–16 is a common sweet spot. Raise it if the task is far from the base model's training.
Quantization for inference (INT8/INT4) is almost free quality-wise and routinely used to fit models on smaller GPUs; push to 4-bit when memory is the binding constraint.
The cost is precision: very aggressive quantization (and very low rank) can degrade quality — measure on your own eval set rather than trusting the smaller number blindly.

Takeaways: LoRA freezes W and learns a low-rank ΔW = BA, training 2rd params instead of d² — often <1% — with swappable, latency-free adapters. Quantization stores weights in fewer bits (FP16/INT8/INT4), cutting memory at the price of small rounding error. QLoRA = 4-bit frozen base + LoRA adapters, so you can fine-tune huge models on a single GPU.