Domain
Efficiency
Ways to make models cheaper to train and serve: quantization, distillation, low-rank adapters, pruning and sparse experts, and the memory, latency and accuracy each one trades.
Preview the recommended route
Start here. Make one prediction, then note what carries over to the next notebook.
WbitsmemorylatencyLearnerWhich notebook should I read first?
Open the first notebookRead the first notebook, make one prediction in its demo, then go on to the next.
Recommended route
Start here, then follow the prerequisites forward.
Read these in order: each notebook comes after the notebooks in this domain that it builds on, and the label beside it gives its level. A prerequisite with no page here yet is marked as outside reading. A notebook is left out of the route only when a prerequisite has neither a page here nor a named outside reading; the full list below marks it.
- 01Knowledge Distillation: Learning from Teachers
Knowledge distillation trains a student, usually smaller, to match a teacher's temperature-softened output probabilities, often alongside the true labels. The probabilities the teacher gives to wrong classes show which classes it treats as similar, which hard labels omit.
Intermediate16 mincodedemoafter Maximum Likelihood, Label Smoothing & Soft Targets - 02Efficiency: Quantization, Distillation, LoRA & Sparse MoE
Four common ways to make large models cheaper to run or adapt: quantize weights and activations, distill a teacher into a smaller student, train a low-rank update (LoRA) or route each token to a few experts (sparse MoE).
Advanced20 mincodedemoafter Maximum Likelihood, Scaled Dot-Product Attention & Transformer Layers, Loss Landscapes, Sharpness & Flat Minima - 03Pruning Weights and Channels
Reduce parameter count by zeroing or removing weights. Unstructured sparsity needs sparse kernels for speed; structured pruning removes whole channels/heads to shrink dense tensor shapes.
Intermediate20 mincodedemoafter Efficiency: Quantization, Distillation, LoRA & Sparse MoE, Weight Initialization: Xavier, He & muPPruning Weights and Channels builds directly on Efficiency.
- 04Quantization: Compressing Models to Integers
Reduce memory and bandwidth by storing weights/activations in low-bit integers (INT8/INT4) with careful scaling to limit accuracy loss.
Intermediate16 mincodedemoafter Efficiency: Quantization, Distillation, LoRA & Sparse MoE, LLM Serving at Scale: Prefill, Decode & Continuous Batching - 05Sparse Mixture of Experts: Routing, Load Balancing & Expert Parallelism
Conditional computation: a router picks a few experts per token. You can increase total expert parameters while keeping activated expert FFN compute small, but distributed systems may pay in communication and scheduling.
Advanced20 mincodedemoafter Scaled Dot-Product Attention & Transformer Layers, Maximum Likelihood, Efficiency: Quantization, Distillation, LoRA & Sparse MoE
All notebooks
Every published notebook in this domain.
Knowledge Distillation: Learning from Teachers
Knowledge distillation trains a student, usually smaller, to match a teacher's temperature-softened output probabilities, often alongside the true labels. The probabilities the teacher gives to wrong classes show which classes it treats as similar, which hard labels omit.
Efficiency: Quantization, Distillation, LoRA & Sparse MoE
Four common ways to make large models cheaper to run or adapt: quantize weights and activations, distill a teacher into a smaller student, train a low-rank update (LoRA) or route each token to a few experts (sparse MoE).
Pruning Weights and Channels
Reduce parameter count by zeroing or removing weights. Unstructured sparsity needs sparse kernels for speed; structured pruning removes whole channels/heads to shrink dense tensor shapes.
Quantization: Compressing Models to Integers
Reduce memory and bandwidth by storing weights/activations in low-bit integers (INT8/INT4) with careful scaling to limit accuracy loss.
Sparse Mixture of Experts: Routing, Load Balancing & Expert Parallelism
Conditional computation: a router picks a few experts per token. You can increase total expert parameters while keeping activated expert FFN compute small, but distributed systems may pay in communication and scheduling.