Continuous Function

Curriculum

A course in modern AI, from first principles to open questions.

Seven parts and 51 notebooks, ordered so that each idea rests on the ones before it. Comfort with algebra and a little programming is enough to begin; later parts assume the earlier ones, and each notebook names its prerequisites.

How each notebook teaches

Every notebook presents one idea four ways. The four views describe the same object with the same numbers, so that a step you cannot follow in one can be checked in another.

  1. 01IntuitionWhat the idea does, in plain words and one picture, before any symbol is introduced.
  2. 02MathematicsThe definitions and the derivation, with every symbol defined and every assumption stated.
  3. 03CodeA short program that computes the same object, on the same numbers, so the equation can be checked line by line.
  4. 04InstrumentA figure you can change. Predict what will happen, change one input, and see which quantities respond.
Foundations. The cubic f(x) = x³ − x and its tangent at x₀ = 0.82. The warm tangent has slope f′(x₀) = 1.0172; its triangle has run Δx = 0.28 and rise Δy = f′(x₀)Δx.fx₀ΔxΔyx

Part I

Mathematical foundations

The dot product as a measure of agreement, the derivative carried backwards through a computation, and probability as the bookkeeping of uncertainty. The part ends with likelihood, cross-entropy and relative entropy, which return later as the losses that models are trained to minimise.

  1. I.1Vector SpacesA vector space is a set of objects that can be added and scaled by numbers, with both operations obeying a few consistent rules.Introductory
  2. I.2Dot ProductThe dot product sums coordinate-wise products and equals the two lengths multiplied by the cosine of the angle between them.Introductory
  3. I.3DerivativesThe derivative is the limit of a secant’s slope as its step shrinks to zero: the instantaneous rate of change at a point.Introductory
  4. I.4Computation GraphsA computation graph writes a calculation as small operations joined by dependencies, so values flow forward and derivatives back.Introductory
  5. I.5Reverse-Mode Automatic DifferentiationReverse mode records the operations that ran, then propagates adjoints backwards to give a scalar’s full gradient in one sweep.Intermediate
  6. I.6BackpropagationBackpropagation carries the chain rule backwards from the loss, reusing each node’s gradient so one pass reaches every weight.Intermediate
  7. I.7Probability BasicsProbability assigns mass to sets of outcomes; conditioning keeps only the outcomes the evidence allows and renormalises.Introductory
  8. I.8Random VariablesA random variable is a function that assigns a number to each outcome; its randomness comes only from which outcome occurs.Introductory
  9. I.9DistributionsA distribution says how probability is spread over the values a random variable can take, as point masses or as a density.Introductory
  10. I.10Maximum LikelihoodMaximum likelihood holds the data fixed and picks the parameters that make them most probable, minimising negative log-likelihood.Intermediate
  11. I.11Bayesian InferenceBayesian inference multiplies a prior by the likelihood and normalises, giving a distribution over the unknowns, not one estimate.Intermediate
  12. I.12Cross-EntropyCross-entropy is a model’s expected surprise under the true distribution: that distribution’s entropy plus the KL divergence.Intermediate
  13. I.13KL DivergenceKL divergence is the expected extra log-loss of using Q where P is true; it is zero only when they agree, and is not symmetric.Intermediate

Guided practice

Moving the camera or moving the object? — Four landmarks through a change of coordinate frame, in three dimensions and on the camera image, with the classic inverted-transform error beside the correct one.

Optimization. Ten gradient-descent steps on L(θ) = ½θᵀAθ, where A has eigenvalues 1 and 10 and axes rotated by −0.38 radians. A warm path with step size 0.18 zigzags toward the minimum through exact quadratic level sets.θ₀θ*η = 0.18

Part II

Learning and optimization

Gradient descent and the adaptive methods used in practice, what the shape of the loss landscape suggests about stability and generalization, and how schedules for the step size are designed around it.

  1. II.1Gradient DescentGradient descent repeatedly steps against the gradient, scaled by a learning rate, using only the local slope to lower the loss.Introductory
  2. II.2Adam OptimizerAdam divides a running average of each parameter’s gradient by the root of a running average of its square, both bias-corrected.Intermediate
  3. II.3Loss Landscapes, Sharpness & Flat MinimaCurvature near a minimum measures its sharpness, a sensitivity to weight changes that also depends on how the weights are scaled.Intermediate
  4. II.4Learning Rate SchedulesA schedule varies the learning rate over training: usually a short warmup from small steps, then a decay, often along a cosine.Intermediate
The transformer. A six-by-six causal attention matrix from fixed scores. Square area is proportional to the row-normalized softmax weight; every row sums to one. Future positions above the diagonal are masked to zero. The fifth row is warm.keyquerymaskedarea ∝ weight

Part III

The transformer

Scaled dot-product attention and the residual block first; then what the learned vectors represent, how text becomes tokens, and the normalization and position schemes that make deep stacks trainable. What attention costs in memory, and how that cost is engineered down, is taken up in Part VI.

  1. III.1Scaled Dot-Product Attention & Transformer LayersScaled dot products of a query with every key, passed through a softmax, give the weights for averaging the values.Intermediate
  2. III.2Representation Learning & Embedding GeometryA learned representation maps inputs to vectors whose distances and directions can encode meaning, context and similarity.Intermediate
  3. III.3Tokenization & Vocabulary DesignByte-pair encoding builds a vocabulary by repeatedly merging the most frequent adjacent pair; text is then read as token IDs.Intermediate
  4. III.4Layer Normalization & RMSNormLayerNorm centres each token’s features and divides by their standard deviation; RMSNorm divides by the root mean square alone.Intermediate
  5. III.5Rotary Position EmbeddingsRoPE rotates query and key coordinate pairs by angles proportional to position, so scores depend on position only via the offset.Intermediate

Guided practice

The transformer lab — Eight stations that rebuild a transformer block by prediction and calculation.

Generative models. Forward diffusion of an equal mixture of N(−2, 0.42²) and N(2, 0.42²). The warm curve is the data density; signal retention ᾱ = 0.6, 0.18 and 0.0025 gives successively noisier densities under xₜ = √ᾱ x₀ + √(1−ᾱ) ε, with independent standard normal ε. The faint final curve is near N(0,1).data≈ N(0, 1)−202x

Part IV

Generative modelling

Latent-variable models and the evidence lower bound, invertible flows with exact densities, and the score-based, diffusion and flow-matching methods behind modern image and video generation, presented as one family of ideas.

  1. IV.1Variational AutoencodersA variational autoencoder maximises a lower bound on log-likelihood; the gap is the KL divergence between encoder and posterior.Advanced
  2. IV.2Normalizing FlowsA normalizing flow warps a simple density through an invertible map; the change-of-variables formula gives its exact likelihood.Advanced
  3. IV.3Diffusion, Score-Based Models & Flow MatchingDiffusion adds Gaussian noise to data step by step and trains a network to undo it, so that sampling can turn noise into data.Advanced
  4. IV.4Score Matching & Score-Based Generative ModelsThe score, the gradient of the log-density, does not involve the normalising constant and can be estimated by learning to denoise.Advanced
  5. IV.5Flow Matching & Rectified FlowsFlow matching regresses a velocity field onto known velocities along paths from noise to data, then integrates it to draw samples.Advanced
Scale. Illustrative loss power law L(C) = 0.6 + 2.4 C^(−0.25), where C is dimensionless compute. Both axes are logarithmic. Warm example points lie on the law; the dashed line is the irreducible floor E = 0.6. These are illustrative values, not measured scaling results.LCE110³10⁶31

Part V

Scale and generalization

The double-descent curve, empirical scaling laws and what they do and do not predict, the infinite-width limit of the neural tangent kernel, and how additional computation at inference time can be traded for accuracy.

  1. V.1Overparameterization & GeneralizationTest error can peak where a model first fits its training data exactly, then fall again as capacity grows beyond that threshold.Intermediate
  2. V.2Scaling Laws & Emergent AbilitiesTest loss falls empirically as a power law in parameters, data and compute, so small runs can forecast larger ones.Intermediate
  3. V.3Neural Tangent Kernel (NTK) & Infinite-Width LimitsAt infinite width a network stays near its linearisation at initialisation and trains like kernel regression with a fixed kernel.Advanced
  4. V.4Test-Time ComputeSampling more answers helps only if some sample is correct and the verifier selects it; coverage and selection fail separately.Advanced
Systems. The signal tanh(x) and its warm three-bit uniform quantization Q(tanh(x)) to eight levels from −1 to 1. The marked spacing is Δ = 2/7; the absolute quantization error is at most Δ/2 for inputs in [−1,1].Δ3 bits · 8 levelsx−1

Part VI

Efficient training and inference

How generation runs, what attention costs in memory through the key–value cache, and how serving systems schedule prefill and decode; then the methods that make it cheaper, each with the error it introduces: quantization, pruning, distillation and sparse mixtures of experts. The part continues into long contexts and FlashAttention, and ends with speculative and structured decoding.

  1. VI.1EfficiencyFour moves make models cheaper at different costs: fewer bits per number, a smaller student, a low-rank update or sparse experts.Advanced
  2. VI.2Decoding & SamplingTemperature rescales logits before the softmax; top-p samples from the smallest set of tokens whose total probability reaches p.Intermediate
  3. VI.3Efficient Attention at ScaleDecoding reads past keys and values from a cache; grouped-query attention shrinks it by letting query heads share key–value heads.Advanced
  4. VI.4LLM Serving at ScaleServing splits into a compute-bound prefill of the prompt and a memory-bound decode, one token at a time, scheduled in batches.Advanced
  5. VI.5QuantizationQuantization stores weights and activations as low-bit integers with a scale, saving memory traffic at the cost of rounding error.Intermediate
  6. VI.6PruningPruning zeroes single weights or removes whole channels and heads; only the latter makes the dense tensors themselves smaller.Intermediate
  7. VI.7Knowledge DistillationDistillation trains a smaller student to match a teacher’s softened output distribution, which says more than the label alone.Intermediate
  8. VI.8Sparse Mixture of ExpertsA router sends each token to a few experts, so total parameters can grow while the computation spent on each token stays small.Advanced
  9. VI.9Long Context EngineeringLong context meets two limits: positions beyond those seen in training, and a key–value cache whose memory grows with each token.Advanced
  10. VI.10FlashAttentionComputes exact attention in tiles held in fast memory, trading extra arithmetic for far fewer reads and writes of slow memory.Advanced
  11. VI.11Speculative DecodingA small model drafts tokens, the large model checks them in parallel, and rejection sampling keeps the output distribution exact.Advanced
  12. VI.12Structured DecodingMasking every token that cannot lead to a valid completion guarantees well-formed output, though not that its content is true.Advanced
  13. VI.13MoE Serving & SchedulingServing sparse experts is a scheduling problem: uneven routing creates stragglers, and moving tokens between devices costs time.Advanced

Guided practice

Attention and serving — A guided route from attention through the cache to batched serving.

Alignment. Reward over-optimization with d = √KL, the square root of policy KL divergence. Gold reward R(d) = d(1 − 0.6 log d), with R(0) = 0, uses the RL form of Gao, Schulman and Hilton (2022) with illustrative coefficients, not measured rewards. Its warm maximum is at d* = exp(1/0.6 − 1). The rising proxy is schematic; the empirical gold form is not reliable near the origin.rewardproxygoldd*d = √KL

Part VII

Alignment, reasoning and interpretability

Reward models and reinforcement learning from human feedback, direct preference methods that skip the reward model, and the ways a proxy reward can be over-optimised; then step-level verifiers and the search over reasoning steps they guide. Sparse autoencoders close the part by asking what a network represents internally.

  1. VII.1RLHFRLHF fits a reward model to human comparisons, then moves the policy towards high reward with a KL penalty to a reference.Advanced
  2. VII.2Direct Preference OptimizationDPO fits preference pairs with a logistic loss on log-ratios to a reference model, without a separate reward model or an RL loop.Advanced
  3. VII.3Kahneman-Tversky OptimizationKTO learns from unpaired outputs labelled good or bad, pushing each implied reward above or below a reference point.Advanced
  4. VII.4Reward HackingOptimising hard against a flawed reward model moves probability onto its errors, so proxy reward can rise as true quality falls.Advanced
  5. VII.5Process Reward ModelsA process reward model scores each reasoning step, not only the final answer: denser feedback that is still a learned proxy.Advanced
  6. VII.6Tree Search ReasoningTree search spends inference budget on partial reasoning, expanding the prefixes a step-level verifier scores as most promising.Advanced
  7. VII.7Sparse AutoencodersA sparse autoencoder approximates an activation with a few directions from a learned dictionary, to separate superposed features.Advanced
Research. Ordinary least squares fitted to six synthetic observations. Past the last observed x, the fit becomes a dashed extrapolation in a tinted unknown region. A pointwise 95% prediction interval for one future observation widens away from the data mean. It assumes a linear mean and independent normal errors with common variance; it does not cover failure of the linear model.observednot yet known95% prediction intervalx

Part VIII

Research

Each part ends at something that is not yet settled. Research here starts from such a question, written down with what would count against it before any study is run.

Open questions The lab

Reference