Domain Neighborhood
Efficiency
How we make models cheaper to train and serve: quantization, distillation, low-rank adapters, sparsity, and the memory/latency tradeoffs that dominate real deployments.
Start here. Predict once, then carry the invariant forward.
WbitsmemorylatencyLearner lensWhat makes this domain feel navigable?
Take this lensStabilize the first mechanism, make one prediction, then move one node forward.
Recommended Route
Start here, then follow the prerequisites forward.
This sequence is ordered for learning rather than inventory. Published notebooks with an unavailable prerequisite—or a same-domain route step that depends on one—are labeled in the full inventory instead of being presented as ready steps.
- 01Knowledge Distillation: Learning from Teachers
Train a smaller student to mimic a stronger teacher by matching soft probability distributions (often with temperature), transferring 'dark knowledge' beyond hard labels.
16 mincodedemoafter Maximum Likelihood, Label Smoothing & Soft TargetsCheck Maximum Likelihood first if the symbols feel slippery.
- 02Efficiency: Quantization, Distillation, LoRA & Sparse MoE
The practical toolkit for making big models cheaper: quantize weights/activations, distill teachers into students, adapt with low-rank updates (LoRA), and use sparsity (MoE).
20 mincodedemoafter Maximum Likelihood, Scaled Dot-Product Attention & Transformer Layers, Loss Landscapes, Sharpness & Flat MinimaWhy this follows: both pages keep the efficiency / distillation thread active.
- 03Pruning: Removing Unnecessary Weights
Reduce parameter count by zeroing or removing weights. Unstructured sparsity needs sparse kernels for speed; structured pruning removes whole channels/heads to shrink dense tensor shapes.
16 mincodedemoafter Efficiency: Quantization, Distillation, LoRA & Sparse MoE, Weight Initialization: Xavier, He & muPWhy this follows: Pruning: Removing Unnecessary Weights uses Efficiency: Quantization, Distillation, LoRA & Sparse MoE directly.
- 04Quantization: Compressing Models to Integers
Reduce memory and bandwidth by storing weights/activations in low-bit integers (INT8/INT4) with careful scaling to limit accuracy loss.
16 mincodedemoafter Efficiency: Quantization, Distillation, LoRA & Sparse MoE, LLM Serving at Scale: Prefill, Decode & Continuous BatchingWhy this follows: both pages keep the efficiency thread active.
- 05Sparse Mixture of Experts: Routing, Load Balancing & Expert Parallelism
Conditional computation: a router picks a few experts per token. You can increase total expert parameters while keeping activated expert FFN compute small, but distributed systems may pay in communication and scheduling.
20 mincodedemoafter Scaled Dot-Product Attention & Transformer Layers, Maximum Likelihood, Efficiency: Quantization, Distillation, LoRA & Sparse MoEWhy this follows: both pages keep the efficiency thread active.
All Published Notebooks
Browse the territory.
Knowledge Distillation: Learning from Teachers
Train a smaller student to mimic a stronger teacher by matching soft probability distributions (often with temperature), transferring 'dark knowledge' beyond hard labels.
Efficiency: Quantization, Distillation, LoRA & Sparse MoE
The practical toolkit for making big models cheaper: quantize weights/activations, distill teachers into students, adapt with low-rank updates (LoRA), and use sparsity (MoE).
Pruning: Removing Unnecessary Weights
Reduce parameter count by zeroing or removing weights. Unstructured sparsity needs sparse kernels for speed; structured pruning removes whole channels/heads to shrink dense tensor shapes.
Quantization: Compressing Models to Integers
Reduce memory and bandwidth by storing weights/activations in low-bit integers (INT8/INT4) with careful scaling to limit accuracy loss.
Sparse Mixture of Experts: Routing, Load Balancing & Expert Parallelism
Conditional computation: a router picks a few experts per token. You can increase total expert parameters while keeping activated expert FFN compute small, but distributed systems may pay in communication and scheduling.
Advanced Bridges