Domain
LLM Systems
How language models are served: prefill and decode, KV-cache memory, batching and scheduling, decoding rules, and techniques such as speculative decoding that cut latency.
Preview the recommended route
Start here. Make one prediction, then note what carries over to the next notebook.
tokensKVbatchlatencyLearnerWhich notebook should I read first?
Open the first notebookRead the first notebook, make one prediction in its demo, then go on to the next.
Recommended route
Start here, then follow the prerequisites forward.
Read these in order: each notebook comes after the notebooks in this domain that it builds on, and the label beside it gives its level. A prerequisite with no page here yet is marked as outside reading. A notebook is left out of the route only when a prerequisite has neither a page here nor a named outside reading; the full list below marks it.
- 01Decoding & Sampling: Temperature, Top-p & Inference-Time Control
How decoding settings reshape the next-token distribution before a token is drawn: temperature, nucleus (top-p) sampling, and why the decoding rule changes what a model writes.
Intermediate16 mincodedemoafter Maximum Likelihood, Scaled Dot-Product Attention & Transformer Layers - 02LLM Serving at Scale: Prefill, Decode & Continuous Batching
How LLM inference is served: prefill and decode, time to first token and time per output token, batching and scheduling, and why KV-cache memory often limits batch size.
Advanced22 mincodedemoafter Scaled Dot-Product Attention & Transformer Layers, Efficient Attention at Scale: KV Cache, GQA & FlashAttention - 03Structured Decoding: Token Masks From Schema Automata
How a schema automaton or parser state turns next-token logits into constraint-valid generation by masking invalid continuations, while leaving truth and task success outside the formal guarantee.
Advanced22 mincodedemoafter Decoding & Sampling: Temperature, Top-p & Inference-Time Control, Tokenization & Vocabulary Design - 04Speculative Decoding: Lossless Multi-Token Generation
Draft several tokens with a fast model, score all the draft prefixes with the target model in one parallel pass, then use modified rejection sampling so the output distribution matches the target model's.
Advanced18 mincodedemoafter Maximum Likelihood, Scaled Dot-Product Attention & Transformer Layers, LLM Serving at Scale: Prefill, Decode & Continuous Batching - 05MoE Serving & Scheduling: Token Dispatch, All-to-All, Disaggregated Parallelism
Serving a mixture-of-experts model turns sparse computation into a scheduling problem: uneven routing creates stragglers and sending tokens to experts costs communication, which motivates careful scheduling and, in systems such as MegaScale-Infer, separate GPU pools for attention and experts.
Advanced20 mincodedemoafter Scaled Dot-Product Attention & Transformer Layers, Efficient Attention at Scale: KV Cache, GQA & FlashAttention, LLM Serving at Scale: Prefill, Decode & Continuous Batching
All notebooks
Every published notebook in this domain.
Decoding & Sampling: Temperature, Top-p & Inference-Time Control
How decoding settings reshape the next-token distribution before a token is drawn: temperature, nucleus (top-p) sampling, and why the decoding rule changes what a model writes.
LLM Serving at Scale: Prefill, Decode & Continuous Batching
How LLM inference is served: prefill and decode, time to first token and time per output token, batching and scheduling, and why KV-cache memory often limits batch size.
Structured Decoding: Token Masks From Schema Automata
How a schema automaton or parser state turns next-token logits into constraint-valid generation by masking invalid continuations, while leaving truth and task success outside the formal guarantee.
Speculative Decoding: Lossless Multi-Token Generation
Draft several tokens with a fast model, score all the draft prefixes with the target model in one parallel pass, then use modified rejection sampling so the output distribution matches the target model's.
MoE Serving & Scheduling: Token Dispatch, All-to-All, Disaggregated Parallelism
Serving a mixture-of-experts model turns sparse computation into a scheduling problem: uneven routing creates stragglers and sending tokens to experts costs communication, which motivates careful scheduling and, in systems such as MegaScale-Infer, separate GPU pools for attention and experts.