Machine Learning

Practical Training Loop

A training loop is a reproducible contract: batches enter, losses produce gradients, the optimizer updates state, validation measures without training, and checkpoints preserve enough state to resume.

status: reviewimportance: criticaldifficulty 3/5math: graduateread: 20mlive demo

Back to Machine Learning Next: Tiny LM Training Loop (review)

Concept Structure

Practical Training Loop

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

3prerequisites

3next concepts

4related links

Learner Contract

What this page should let you do.

You are here becauseA training loop is a reproducible contract: batches enter, losses produce gradients, the optimizer updates state, validation measures without training, and checkpoints preserve enough state to resume.

This Machine Learning concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

Before thisMLPs and Universal Approximation (review)

3 prerequisites listed; refresh them before leaning on the math or code.

MLPs and Universal Approximation (review)Backpropagation Weight Decay & AdamW: Decoupled Regularization (review)

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Then go nextTiny LM Training Loop (review)

Follow this edge after making one prediction here; the next page should reuse the result, not restart the route.

Test the linkManipulate one control and predict the visible change.Then continue to Tiny LM Training Loop (review)

Claim/source review status

Substantive review recorded

1/1 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.

Claims1/1 reviewed

Sources6 cited

Codeattached

Demolive

Reviewed2026-07-02

Updatedpage 2026-07-02

Learning item flow

4/4 sections readyAsk about this Research room

ConceptPractical Training LoopMachine Learning EquationPractical Training Loop equation 1Exact equation CodePractical Training Loop runnable code 1Exact runnable code DemoPractical Training Loop interactive demoInteractive demo ClaimA practical neural-network training loop is a contract over minibatch...Exact claim check SourceDive into Deep Learning: Linear Regression Implementation from ScratchExact source

ConceptPractical Training LoopMachine Learning

6 sources attachedLocal snapshot ready

concept:machine-learning/practical-training-loop

Codenearby Predictbefore reveal Roomitem handoff

01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

The training loop is where a neural network stops being an equation on a page and becomes an experiment you can reproduce. A good loop is not just "call fit." It is a contract: every batch must enter with the right shape and device, the model must be in the right mode, the loss must create the right scalar objective, gradients must be fresh, the optimizer must update parameters and its own state, validation must measure without accidentally training, and a checkpoint must preserve enough state to resume honestly.

That contract is why this page belongs before from-scratch Transformer work. If you cannot inspect one iteration, you cannot debug a 100-million-parameter run. The details get bigger later, but the grammar stays recognizable:

Batch: a Dataset defines samples, and a DataLoader turns them into minibatches.
Mode: train() lets training-time layers behave as training-time layers; eval() changes dropout and normalization behavior for measurement.
Forward and loss: the model produces predictions, and the loss compresses a minibatch into one scalar.
Gradient reset: accumulated gradients are useful in special microbatch setups, but the default contract resets them before the new backward pass.
Backward: reverse-mode autodiff deposits parameter gradients.
Optimizer step: the optimizer uses gradients and optimizer state to update parameters.
Validation: measurement should usually run under eval() and no_grad().
Checkpoint: resuming training needs more than model weights if the optimizer, scheduler, scaler, RNG state, epoch, and config matter.

Mixed precision is a boundary, not magic dust. In a typical PyTorch AMP loop, autocast wraps the forward and loss computation, while scaling/unscaling and stepping live around backward and optimizer update. If you save a scaler or scheduler, it becomes part of the checkpoint contract.

The practical habit to build is simple: when a loss trace surprises you, do not start with folklore. Ask which part of the contract was broken.

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt

Equation 1 $L_t(\theta) = \frac{1}{|\mathcal B_t|} \sum_{i\in\mathcal B_t} \ell(f_\theta(x_i), y_i).$ Equation 2 $g_t = \nabla_\theta L_t(\theta_t).$

Let a training set contain examples $(x_i, y_i)$ , and let the model $f_\theta$ have parameters $\theta$ . For a minibatch $\mathcal B_t$ at step $t$ , the loop estimates the empirical-risk gradient by averaging losses over that batch:

L_t(\theta) = \frac{1}{|\mathcal B_t|} \sum_{i\in\mathcal B_t} \ell(f_\theta(x_i), y_i).

The backward pass computes

g_t = \nabla_\theta L_t(\theta_t).

A plain SGD step would be

\theta_{t+1} = \theta_t - \eta_t g_t,

where $\eta_t$ is the learning rate. A practical optimizer such as AdamW is better viewed as a stateful update:

(\theta_{t+1}, s_{t+1}) = \mathrm{OptimizerUpdate}(\theta_t, s_t, g_t, h_t),

where $s_t$ is optimizer state such as moments, and $h_t$ contains hyperparameters such as learning rate and weight decay. That is why checkpointing only $\theta_t$ is not the same as checkpointing a resumable training run.

A serious checkpoint is a tuple like

C_t = (\theta_t,\ s_t,\ q_t,\ a_t,\ r_t,\ t,\ m_t,\ c),

where $q_t$ may include scheduler state, $a_t$ may include mixed-precision scaler state, $r_t$ is RNG/data-order state when exact continuation matters, $m_t$ stores metrics, and $c$ stores the run config. Different projects need different pieces, but the question is always the same: if you restart from $C_t$ , what hidden state did you silently reset?

Validation uses a separate measurement objective, often

\hat L_{\mathrm{val}}(\theta_t) = \frac{1}{|\mathcal V|} \sum_{i\in\mathcal V} \ell(f_{\theta_t}(x_i), y_i),

computed without parameter updates. This number is not a gradient step. It is a measurement of the current model under validation-mode behavior.

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt

Runnable code 1import torch from torch import nn device = "cuda" if torch.cuda.is_available() else "cpu" mod...python

import torch
from torch import nn

device = "cuda" if torch.cuda.is_available() else "cpu"
model = nn.Sequential(nn.Linear(784, 128), nn.ReLU(), nn.Linear(128, 10)).to(device)
loss_fn = nn.CrossEntropyLoss()
optimizer = torch.optim.AdamW(model.parameters(), lr=3e-4, weight_decay=0.01)

def run_epoch(loader, training: bool):
    model.train(training)
    total = 0.0
    for xb, yb in loader:
        xb, yb = xb.to(device), yb.to(device)
        with torch.set_grad_enabled(training):
            logits = model(xb)
            loss = loss_fn(logits, yb)
            if training:
                optimizer.zero_grad(set_to_none=True)
                loss.backward()
                optimizer.step()
        total += loss.item() * xb.size(0)
    return total / len(loader.dataset)

for epoch in range(5):
    train_loss = run_epoch(train_loader, training=True)
    val_loss = run_epoch(val_loader, training=False)
    torch.save({
        "epoch": epoch,
        "model": model.state_dict(),
        "optimizer": optimizer.state_dict(),
        "train_loss": train_loss,
        "val_loss": val_loss,
        "config": {"lr": 3e-4, "weight_decay": 0.01},
    }, "checkpoint.pt")

This is intentionally small. Real systems may add gradient accumulation, clipping, schedulers, mixed precision, distributed wrappers, logging, early stopping, and artifact versioning. The invariant is still inspectable: which state was read, which scalar was differentiated, which state was updated, and which state was saved?

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Use the Practical Training Loop Lab as a diagnostic prediction check.

Choose a failure trace and commit to the missing contract step before reveal:

Forgot zero_grad: gradients accumulate across batches when the run intended one batch per update.
Skipped optimizer.step: backward computes gradients, but parameters never move.
Stayed in train() during validation: measurement is contaminated by training-time layer behavior.
Saved a weak checkpoint: model weights resume, but optimizer/scheduler/scaler/RNG state silently resets.

Before reveal, the answer-bearing stage, loss trace values, update norms, and checkpoint payload verdicts stay hidden. After reveal, inspect the highlighted contract stage, the toy trace, the checkpoint ledger, and the concrete reason the symptom follows from the broken step.

This is a diagnostic toy, not a full training framework. Its purpose is to make future labs easier to trust: when a larger model misbehaves, you have a compact checklist before you reach for guesses.

Live Concept Demo

Explore Practical Training Loop

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 3/5graduatecode-aligned

Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what Practical Training Loop should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

Prediction open01 / Intuition

Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Practical Training Loop should make visible.

Visual Inquiry

Make the image answer a mathematical question

4/4 stages readyLive demo connected

Visual cueWhich visible object should carry the first intuition?

Inspection depth2/4

Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Practical Training Loop easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

book · 2026Dive into Deep Learning: Linear Regression Implementation from ScratchZhang, Lipton, Li, and Smola

Supports the epoch/iteration/minibatch training loop, train/eval mode split, gradient computation, zeroing, and optimizer step sequence.

Open source

book · 2026Dive into Deep Learning: Minibatch Stochastic Gradient DescentZhang, Lipton, Li, and Smola

Supports the minibatch empirical-risk gradient and the computational/statistical reason for batching examples.

Open source

documentation · 2026PyTorch Tutorials: Datasets and DataLoadersPyTorch

Supports Dataset/DataLoader separation, minibatches, shuffling, and one-batch iteration shape checks.

Open source

documentation · 2026PyTorch Tutorials: Optimizing Model ParametersPyTorch

Supports train_loop/test_loop structure, model.train(), model.eval(), torch.no_grad(), zero_grad, loss.backward, and optimizer.step.

Open source

documentation · 2026PyTorch Tutorials: Saving and Loading ModelsPyTorch

Supports state_dict, checkpoint dictionaries, saving optimizer state for resuming, and train/eval mode after load.

Open source

documentation · 2026PyTorch Documentation: Automatic Mixed Precision package - torch.ampPyTorch

Supports autocast as a forward/loss-region context and GradScaler as the usual float16 AMP training companion.

Open source

Claim Review

A training loop is a reproducible contract: batches enter, losses produce gradients, the optimizer updates state, validation measures without training, and checkpoints preserve enough state to resume.

Status1 substantive review recorded

Claims without a substantive review badge still need exact source-support review.

Sources6 references

d2l-linear-regression-scratch-training, d2l-minibatch-sgd, pytorch-dataloaders, pytorch-optimization-loop, pytorch-saving-loading, pytorch-amp

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedA practical neural-network training loop is a contract over minibatch data, train/eval mode, forward loss, gradient reset, backward gradient computation, optimizer state update, validation without gradient accumulation, and checkpoint state for reproducible resume.Claim metadata: source checked

The sources jointly support minibatch objective estimates, DataLoader batching/shuffling, train and validation loop separation, zero/backward/step mechanics, state_dict checkpointing for resume, and mixed-precision forward/loss boundary caveats.

Sources: Dive into Deep Learning: Linear Regression Implementation from Scratch, Dive into Deep Learning: Minibatch Stochastic Gradient Descent, PyTorch Tutorials: Datasets and DataLoaders, PyTorch Tutorials: Optimizing Model Parameters, PyTorch Tutorials: Saving and Loading Models, PyTorch Documentation: Automatic Mixed Precision package - torch.ampToy diagnostic lab only; not distributed training, gradient accumulation over microbatches, DDP/FSDP, optimizer implementation internals, scheduler design, exact deterministic replay on every hardware stack, or GPT Pro publication approval.A bounded review summary is present; still check caveats and exact reference scope.

Checked D2L for minibatch gradients and train/eval loop structure; PyTorch docs for DataLoader batching, zero_grad/backward/step, test_loop/no_grad, checkpoint state_dict resume, and AMP boundaries. Stanford CS336 supports the downstream from-scratch Transformer training motivation. GPT Pro/Oracle publication critique is still pending.

Reviewer: codex-local-primary-source-audit; reviewed 2026-07-02

source-span-d2l-linear-regression-scratch-training source-span-d2l-minibatch-sgd source-span-pytorch-dataloaders source-span-pytorch-optimization-loop source-span-pytorch-saving-loading Equation 1 Runnable code 1 Interactive demo

Source support candidates

book 2026Dive into Deep Learning: Linear Regression Implementation from Scratch

Supports the epoch/iteration/minibatch training loop, train/eval mode split, gradient computation, zeroing, and optimizer step sequence.

book 2026Dive into Deep Learning: Minibatch Stochastic Gradient Descent

Supports the minibatch empirical-risk gradient and the computational/statistical reason for batching examples.

documentation 2026PyTorch Tutorials: Datasets and DataLoaders

Supports Dataset/DataLoader separation, minibatches, shuffling, and one-batch iteration shape checks.

documentation 2026PyTorch Tutorials: Optimizing Model Parameters

Supports train_loop/test_loop structure, model.train(), model.eval(), torch.no_grad(), zero_grad, loss.backward, and optimizer.step.

Mechanism checks

Equation 1

L_t(\theta) = \frac{1}{|\mathcal B_t|} \sum_{i\in\mathcal B_t} \ell(f_\theta(x_i), y_i).

Equation 2

g_t = \nabla_\theta L_t(\theta_t).

Runnable code 1import torch from torch import nn device = "cuda" if torch.cuda.is_available() else "cpu" mod...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

Readiness0/3 checks ready

Predict

Before touching the demo, predict one visible change that should happen in Practical Training Loop.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Your answer canvas

Local checks

Claim

A concrete answer is on the canvas.

Mechanism

The answer names why the claim should hold.

Bridge

It touches the page context or a neighboring idea.

Misconception check

Grounded research drawerClose

ConceptPractical Training LoopMachine Learning

Runnable code comparisonPractical Training Loop runnable code 1device = "cuda" if torch.cuda.is_available() else "cpu"Prediction before revealPractical Training Loop interactive demoManipulate one control and predict the visible change.

Grounded room questionWhat is the smallest example that makes Practical Training Loop click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptMachine Learning

Practical Training Loop

Attached question

What is the smallest example that makes Practical Training Loop click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action

Local action draft

This draft stays in this browser, attached to the selected learning item.

Draft noteNext action

No local draft saved.

Evidence to inspect

References to inspect: attached references on this page.
Definition, prerequisite, and contrast concept links
The equation or runnable code that makes the concept operational
One demo state that shows the invariant instead of a slogan

What would resolve this

The learner can state the mechanism in their own words
The learner can name the prerequisite that would repair confusion
The learner can predict how the mechanism changes under one perturbation

Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - Practical Training Loop Selected item key: recorded for copy. Context: Machine Learning Page anchor: recorded for copy. Open question: What is the smallest example that makes Practical Training Loop click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context

concept/concept-notebook/machine-learning/practical-training-loop
concept:machine-learning/practical-training-loop

Learning Map

Before / Now / Try / Next

BeforeMLPs and Universal Approximation (review)

NowIntuition -> Math -> Code -> Demo

TryManipulate one control and predict the visible change.

NextTiny LM Training Loop (review)

Intuitionready
Mathready
Codeready
Interactive Demoready

AI Companion

Ask beside the selected object

Your question

GoalComfortStyleStuck on

Context prompt

You are my AI learning companion for Continuous Function. Current context: Machine Learning concept. Learning surface: Practical Training Loop. What this page says: A training loop is a reproducible contract: batches enter, losses produce gradients, the optimizer updates state, validation measures without training, and checkpoints preserve enough state to resume. Current section: Intuition, math, code, and interactive demo. Suggested next step: Manipulate one control and predict the visible change.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Explain the central idea in plain language, then restate it with the exact math objects from the page. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.

Domain

Machine Learning

machine-learningdeep-learningtrainingoptimizationcheckpoints

Prerequisites

MLPs and Universal Approximation (review)Backpropagation Weight Decay & AdamW: Decoupled Regularization (review)

Leads To

Tiny LM Training Loop (review)Transformer Implementation From Scratch (planned)Training Loss Curves and Debugging (review)

Gradient Descent Adam Optimizer Gradient Clipping & Explosion Prevention (review)BatchNorm, LayerNorm, and RMSNorm (review)

Within this domain