Continuous Function

Foundation reference

Efficiency: Quantization, Distillation, LoRA & Sparse MoE

Quantization stores weights with fewer bits, and LoRA trains a low-rank update while keeping the original weights fixed. Combining 4-bit quantization with LoRA makes it possible to fine-tune a 65B-parameter model on a single 48 GB GPU (QLoRA, Dettmers et al., 2023).

A newer notebook covers this concept. This older page keeps its summary, math and sources.

Concept 16 of 100EfficiencyStudy phase 8
Open the demoOpen the newer notebookAll concepts
Key equation
W′=W+BA,r≪dW' = W + BA,\quad r \ll d
Reading map and next steps

Study guide

Where to start

Concept 16 of 100EfficiencyEfficiency · Study phase 8: Scaling, theory & multimodal
Question

What does a low-rank update look like geometrically, and why can a rank-r change to each weight matrix be enough to adapt a pretrained model?

Open the demoGo to the mathOpen the newer notebookNext concept
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.

Before you move a control in the demo, say what you expect to change.

CheckCheck against the math and the papers.

Compare what the demo shows with the key equation and with the papers cited below.

GeneralizeName what stays the same.

Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.

ContinueContinue in the newer notebook.

The newer notebook covers the same concept and has an interactive demo.

Why It Matters for Modern Models

  • Quantization stores weights with fewer bits, and LoRA trains a low-rank update while keeping the original weights fixed. Combining 4-bit quantization with LoRA makes it possible to fine-tune a 65B-parameter model on a single 48 GB GPU (QLoRA, Dettmers et al., 2023).
  • Distillation trains a smaller student to match a larger teacher; DistilBERT is 40% smaller than BERT and keeps about 97% of its language-understanding performance (Sanh et al., 2019).
  • Sparse mixture-of-experts layers raise the parameter count without raising the compute per token; Switch Transformers reached 1.6 trillion parameters this way (Fedus et al., 2022).

Questions to Think About

  • How does quantization error propagate through a network, and why are some layers and outlier channels more sensitive than others?

Interactive Demos

Core Math

If you prefer intuition first, try the demos above, then come back for the math.

Key Equation
W′=W+BA,r≪dW' = W + BA,\quad r \ll d

Distillation: train a student qψq_\psi to match a teacher pθp_\theta, with both softmaxes computed at temperature TT:

L=T2 KL(pθT(⋅∣x) ∥ qψT(⋅∣x))\mathcal L = T^2\,\mathrm{KL}(p_\theta^T(\cdot\mid x)\,\|\,q_\psi^T(\cdot\mid x))

The factor T2T^2 keeps gradient magnitudes comparable across temperatures, and the term is usually combined with the ordinary cross-entropy on the true labels (Hinton et al., 2015).

Quantization: map float weights to a grid of low-bit integers with step Δ\Delta: w~=Δ⋅round(w/Δ)\tilde w = \Delta \cdot \mathrm{round}(w/\Delta)

LoRA: reparameterize a d×dd\times d weight matrix as W′=W+BAW' = W + BA, where B∈Rd×r,A∈Rr×d,r≪dB\in\mathbb R^{d\times r}, A\in\mathbb R^{r\times d}, r\ll d; only AA and BB are trained and WW stays frozen.

Sparse MoE: replace a feed-forward layer by many experts fef_e and a router with probabilities pe(x)p_e(x). With top-1 routing as in Switch Transformers: FFNMoE(x)=pe∗(x) fe∗(x)\text{FFN}_{\text{MoE}}(x) = p_{e^*}(x)\, f_{e^*}(x), where e∗(x)=arg⁡max⁡epe(x)e^*(x) = \arg\max_e p_e(x).

Key Papers

Distilling the Knowledge in a Neural Network

Hinton, Vinyals & Dean2015NIPS 2014 Deep Learning Workshop
Read paper →

LoRA: Low-Rank Adaptation of Large Language Models

Hu et al.2022ICLR
Read paper →

Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

Fedus, Zoph & Shazeer2022JMLR
Read paper →

QLoRA: Efficient Finetuning of Quantized LLMs

Dettmers et al.2023NeurIPS
Read paper →

Connections

Prerequisites

SharpnessDiffusion

Leads to

Efficient AttentionSpeculative DecodingMoEDistributed TrainingQuantizationPruning

Where to go next

A newer notebook covers Efficiency, with an interactive demo.

Continue in the newer notebook
Other connections (6)

Which limitations does it address, or leave to a later idea?

  • Accelerate sampling: a limitation of Diffusion that Efficiency addresses. Open Diffusion

    High-quality diffusion sampling needs tens to hundreds of denoising steps. Distillation methods cut that number: progressive distillation halves the number of steps repeatedly (Salimans & Ho, 2022), and consistency models generate in one or a few steps (Song et al., 2023).

Where does it break down?

  • Sharp minima: where Efficiency breaks down, taken up in Sharpness. Open Sharpness

    Quantization and low-rank approximation perturb the weights (and, for quantized activations, the activations). Where the loss is sharp, small perturbations cause large increases in loss, so compression can degrade quality much more than its size suggests.

  • Smaller reward models: where Efficiency breaks down, taken up in Reward Hacking. Open Reward Hacking

    Cheaper alignment pipelines can be more exposed to reward hacking. Gao et al. (2023) found that smaller proxy reward models are over-optimized more readily than larger ones, so shrinking the reward model to save compute leaves less room to optimize against it safely.

Where else does the same idea appear?

  • KL anchoring: the same technique appears in RLHF. Open RLHF

    Distillation pulls a student toward a teacher by minimizing KL(teacher ‖ student); RLHF adds a penalty KL(policy ‖ reference) that keeps the policy close to the model it started from. Both use a KL divergence to tie one distribution to another, though in opposite directions and for different purposes: in distillation it is the objective, in RLHF a brake on the objective.

  • Student as draft: an analogy between Efficiency and Speculative Decoding. Open Speculative Decoding

    Distillation trains a smaller student to mimic a large teacher. Speculative decoding uses a small "student-like" draft model to propose tokens, then the teacher verifies—same teacher/student pattern at inference time.

  • Adaptive step sizing: the same technique appears in Adam. Open Adam

    Adam's per-parameter step sizes let one learning rate work across parameters whose gradients differ greatly in scale. Parameter-efficient fine-tuning methods such as LoRA and QLoRA are usually trained with Adam-type optimizers (Hu et al., 2022; Dettmers et al., 2023).