Production ML

Experiment Tracking

Experiment tracking records the config, code, data, metric curve, checkpoint, artifacts, and notes that make a run comparable instead of mysterious.

status: reviewimportance: importantdifficulty 3/5math: undergraduateread: 14mlive demo

Back to Production ML Next: Evaluation Pipelines

Concept Structure

Experiment Tracking

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

2prerequisites

2next concepts

3related links

Learner Contract

What this page should let you do.

You are here becauseExperiment tracking records the config, code, data, metric curve, checkpoint, artifacts, and notes that make a run comparable instead of mysterious.

This Production ML concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

Before thisTraining Loss Curves and Debugging (review)

2 prerequisites listed; refresh them before leaning on the math or code.

Training Loss Curves and Debugging (review)Dataset Versioning (review)

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Then go nextEvaluation Pipelines

Follow this edge after making one prediction here; the next page should reuse the result, not restart the route.

Test the linkManipulate one control and predict the visible change.Then continue to Evaluation Pipelines

Claim/source review status

Substantive review recorded

1/1 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.

Claims1/1 reviewed

Sources6 cited

Codeattached

Demolive

Reviewed2026-07-02

Updatedpage 2026-07-02

Learning item flow

4/4 sections readyAsk about this Research room

ConceptExperiment TrackingProduction ML EquationExperiment Tracking equation 1Exact equation CodeExperiment Tracking runnable code 1Exact runnable code DemoExperiment Tracking interactive demoInteractive demo ClaimA useful experiment record connects one run id to config, code versio...Exact claim check SourceFull Stack Deep Learning 2021: Infrastructure and ToolingExact source

ConceptExperiment TrackingProduction ML

6 sources attachedLocal snapshot ready

concept:production-ml/experiment-tracking

Codenearby Predictbefore reveal Roomitem handoff

01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

Experiment tracking is the habit that prevents the sentence every ML team eventually dreads: "Which run was that?"

A loss curve by itself is only a trace. A checkpoint by itself is only a file. A screenshot from a notebook is only a memory. To make a result useful, we need a run record that ties the visible outcome back to the exact choices that produced it: the config, code version, data version, random seed, metric curve, checkpoint, artifacts, and notes.

This page starts deliberately low-tech. You do not need a hosted tracking platform to learn the concept. You need one small rule:

Before comparing two runs, ask whether each run has enough evidence to be rerun or audited.

If Run B has a better validation loss but says only "latest code" and "current data", it is not better evidence than Run A. It is a loose number. If Run C has a worse metric but a complete run record, it may be more valuable for learning because you can explain what changed and reproduce the failure.

Good tracking is not paperwork. It is a way to make learning cumulative.

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt

Equation 1 $R_r = (c_r,\ k_r,\ d_r,\ s_r,\ m_r(t),\ a_r,\ q_r,\ n_r),$ Equation 2 $M(r) < M(u)$

Let a single training run be indexed by a run id $r$ . A minimal run record is

R_r = (c_r,\ k_r,\ d_r,\ s_r,\ m_r(t),\ a_r,\ q_r,\ n_r),

where:

$c_r$ is the configuration: architecture, optimizer, learning rate, batch size, schedule, and other choices.
$k_r$ is the code version, such as a commit hash.
$d_r$ is the data version or dataset evidence bundle.
$s_r$ is the random seed and any other reproducibility seed state worth recording.
$m_r(t)$ is the metric curve over step or epoch $t$ .
$a_r$ is the artifact pointer, such as a log file, plot, report, or exported predictions.
$q_r$ is the checkpoint pointer, usually including enough model/optimizer state to inspect or resume.
$n_r$ is the human note: hypothesis, caveat, bug, or decision.

The record does not prove the run is good. It makes the claim inspectable.

Suppose we compare two runs $r$ and $u$ using a validation metric $M$ . A comparison like

M(r) < M(u)

only answers the intended question when the uncontrolled parts of the record are pinned. For a learning-rate comparison, for example, we might require

d_r = d_u,\quad k_r = k_u,\quad \text{architecture}(c_r) = \text{architecture}(c_u),

and allow only the learning-rate field to differ. If the data version or code version is missing, the comparison is blocked because the metric difference may be caused by an unrecorded change.

A useful audit score can be written as a simple completeness ratio:

A(R_r) = \frac{1}{|F|} \sum_{f\in F} \mathbf 1[f\ \text{is present in}\ R_r],

where $F$ is the required field set for the claim you want to make. This is not a universal production score. It is a teaching device: a low score means the result is hard to compare, even if the loss number looks exciting.

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt

Runnable code 1import csv, hashlib, json, os config = {"model": "tiny_lm", "lr": 3e-4, "batch": 64, "ctx": 1...python

This witness records one toy run as JSON plus a metrics CSV. It is intentionally plain Python so the idea is visible without adopting a tracking server.

import csv, hashlib, json, os

config = {"model": "tiny_lm", "lr": 3e-4, "batch": 64, "ctx": 128}
metrics = [(0, 2.31, 2.28), (100, 1.72, 1.81), (200, 1.31, 1.49)]

def digest(value):
    payload = json.dumps(value, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(payload.encode()).hexdigest()[:12]

run = {
    "run_id": "run_042",
    "config": config,
    "config_hash": digest(config),
    "git_commit": "a1b2c3d",
    "data_version": "reviews-v1.4.2",
    "seed": 42,
    "metrics_path": "metrics.csv",
    "checkpoint": "ckpt_step_0200.pt",
    "artifact": "loss_curve.png",
    "notes": "LR sweep; Run A baseline held fixed.",
}

os.makedirs("runs/run_042", exist_ok=True)
with open("runs/run_042/run.json", "w") as f:
    json.dump(run, f, indent=2, sort_keys=True)

with open("runs/run_042/metrics.csv", "w", newline="") as f:
    writer = csv.writer(f)
    writer.writerow(["step", "train_loss", "val_loss"])
    writer.writerows(metrics)

required = ["config", "git_commit", "data_version", "seed",
            "metrics_path", "checkpoint", "artifact"]
missing = [field for field in required if not run.get(field)]
print("comparable" if not missing else "blocked", missing)

The important part is not the file format. The important part is that the next learner can answer: what changed, what stayed fixed, what metric was measured, what artifact backs the curve, and which checkpoint produced the claim?

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Use the Run Record Lab as an evidence-before-comparison audit.

Before reveal, inspect a sealed run packet and predict which required fields are missing or unknown. Then commit your audit. The lab reveals whether the run has enough evidence for a comparison with the baseline:

Config: the knobs and architecture that define the experiment.
Git commit: the code version behind the result.
Data version: the dataset snapshot and split evidence.
Seed: the random seed used for the run.
Metric curve: the time series, not just the final scalar.
Checkpoint: the state pointer behind the reported best point.
Artifact: the plot/report/predictions that make the claim inspectable.

The rule is simple: a better metric is not enough. If the record is missing code, data, or checkpoint evidence, the comparison is blocked until the record is repaired.

This is a toy audit surface, not a replacement for MLflow, W&B, DVC, governance, or team infrastructure. The point is to learn the run-record contract before outsourcing the habit to tools.

Live Concept Demo

Explore Experiment Tracking

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 3/5undergraduatecode-aligned

Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what Experiment Tracking should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

Experiment tracking records the config, code, data, metric curve, checkpoint, artifacts, and notes that make a run comparable instead of mysterious.

Prediction open01 / Intuition

Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Experiment Tracking should make visible.

Visual Inquiry

Make the image answer a mathematical question

Experiment tracking records the config, code, data, metric curve, checkpoint, artifacts, and notes that make a run comparable instead of mysterious.

4/4 stages readyLive demo connected

Visual cueWhich visible object should carry the first intuition?

Inspection depth2/4

Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Experiment Tracking easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

course-notes · 2021Full Stack Deep Learning 2021: Infrastructure and ToolingFull Stack Deep Learning

Supports treating experiment management as training/evaluation infrastructure, including the need to avoid losing track of code, hyperparameters, metrics, and artifacts across model runs.

Open source

reference · 2023Google Research Tuning PlaybookGoogle Research

Supports configuring experiments with clear goals, tracking studies/trials, selecting checkpoints, and setting up experiment tracking during deep learning tuning.

Open source

documentation · 2026MLflow Documentation: ML Experiment TrackingMLflow

Supports the run-level vocabulary of experiments, runs, parameters, metrics, artifacts, datasets, and model/checkpoint traceability.

Open source

documentation · 2026Weights & Biases Documentation: Experiments OverviewWeights & Biases

Supports logging metrics, hyperparameters/configuration, system metrics, and model artifacts as experiment-tracking records.

Open source

documentation · 2025PyTorch Tutorials: Saving and Loading ModelsPyTorch

Supports checkpoint/state_dict vocabulary for saving model and optimizer state used by resumable and inspectable training runs.

Open source

documentation · 2026DVC Documentation: Data VersioningDVC

Supports versioning data, models, metrics, and artifacts alongside code as reproducibility evidence.

Open source

Claim Review

Experiment tracking records the config, code, data, metric curve, checkpoint, artifacts, and notes that make a run comparable instead of mysterious.

Status1 substantive review recorded

Claims without a substantive review badge still need exact source-support review.

Sources6 references

fsdl-2021-ml-infrastructure-experiment-management, google-tuning-playbook-experiment-tracking, mlflow-tracking, wandb-experiments-overview, pytorch-saving-loading-models, dvc-data-versioning

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedA useful experiment record connects one run id to config, code version, data version, seed, metric curve, checkpoint or artifact pointers, and notes so a reported result can be compared, audited, and rerun.Claim metadata: source checked

FSDL motivates experiment management to avoid losing track of code, hyperparameters, metrics, and artifacts; Google Tuning Playbook motivates disciplined studies, checkpoints, goals, and tracking; MLflow and W&B provide run/config/metric/artifact vocabularies; PyTorch supports checkpoint state; DVC supports versioned data/artifact evidence.

Sources: Full Stack Deep Learning 2021: Infrastructure and Tooling, Google Research Tuning Playbook, MLflow Documentation: ML Experiment Tracking, Weights & Biases Documentation: Experiments Overview, PyTorch Tutorials: Saving and Loading Models, DVC Documentation: Data VersioningThe tuple is a minimal learner-facing contract, not a claim that a JSON file is enough for regulated production, team governance, privacy review, lineage, access control, or large-scale tracking infrastructure.A bounded review summary is present; still check caveats and exact reference scope.

Checked FSDL, Google Research's Tuning Playbook, MLflow tracking docs, W&B experiment docs, PyTorch checkpointing docs, and DVC data-versioning docs. The sources jointly support the run-record fields as a compact teaching contract, while the page stays tool-agnostic and review-status until GPT Pro/Oracle critique is available.

Reviewer: codex-local-primary-source-audit; reviewed 2026-07-02

source-span-fsdl-2021-ml-infrastructure-experiment-management source-span-google-tuning-playbook-experiment-tracking source-span-mlflow-tracking Equation 1 Runnable code 1 Interactive demo

Source support candidates

course-notes 2021Full Stack Deep Learning 2021: Infrastructure and Tooling

Supports treating experiment management as training/evaluation infrastructure, including the need to avoid losing track of code, hyperparameters, metrics, and artifacts across model runs.

reference 2023Google Research Tuning Playbook

Supports configuring experiments with clear goals, tracking studies/trials, selecting checkpoints, and setting up experiment tracking during deep learning tuning.

documentation 2026MLflow Documentation: ML Experiment Tracking

Supports the run-level vocabulary of experiments, runs, parameters, metrics, artifacts, datasets, and model/checkpoint traceability.

documentation 2026Weights & Biases Documentation: Experiments Overview

Supports logging metrics, hyperparameters/configuration, system metrics, and model artifacts as experiment-tracking records.

Mechanism checks

Equation 1

R_r = (c_r,\ k_r,\ d_r,\ s_r,\ m_r(t),\ a_r,\ q_r,\ n_r),

Equation 2

M(r) < M(u)

Runnable code 1import csv, hashlib, json, os config = {"model": "tiny_lm", "lr": 3e-4, "batch": 64, "ctx": 1...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

Experiment tracking records the config, code, data, metric curve, checkpoint, artifacts, and notes that make a run comparable instead of mysterious.

Readiness0/3 checks ready

Predict

Before touching the demo, predict one visible change that should happen in Experiment Tracking.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Your answer canvas

Local checks

Claim

A concrete answer is on the canvas.

Mechanism

The answer names why the claim should hold.

Bridge

It touches the page context or a neighboring idea.

Misconception check

Grounded research drawerClose

ConceptExperiment TrackingProduction ML

Runnable code comparisonExperiment Tracking runnable code 1config = {"model": "tiny_lm", "lr": 3e-4, "batch": 64, "ctx": 128}Prediction before revealExperiment Tracking interactive demoManipulate one control and predict the visible change.

Grounded room questionWhat is the smallest example that makes Experiment Tracking click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptProduction ML

Experiment Tracking

Attached question

What is the smallest example that makes Experiment Tracking click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action

Local action draft

This draft stays in this browser, attached to the selected learning item.

Draft noteNext action

No local draft saved.

Evidence to inspect

References to inspect: attached references on this page.
Definition, prerequisite, and contrast concept links
The equation or runnable code that makes the concept operational
One demo state that shows the invariant instead of a slogan

What would resolve this

The learner can state the mechanism in their own words
The learner can name the prerequisite that would repair confusion
The learner can predict how the mechanism changes under one perturbation

Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - Experiment Tracking Selected item key: recorded for copy. Context: Production ML Page anchor: recorded for copy. Open question: What is the smallest example that makes Experiment Tracking click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context

concept/concept-notebook/production-ml/experiment-tracking
concept:production-ml/experiment-tracking

Learning Map

Before / Now / Try / Next

BeforeTraining Loss Curves and Debugging (review)

NowIntuition -> Math -> Code -> Demo

TryManipulate one control and predict the visible change.

NextEvaluation Pipelines

Intuitionready
Mathready
Codeready
Interactive Demoready

AI Companion

Ask beside the selected object

Experiment tracking records the config, code, data, metric curve, checkpoint, artifacts, and notes that make a run comparable instead of mysterious.

Your question

GoalComfortStyleStuck on

Context prompt

You are my AI learning companion for Continuous Function. Current context: Production ML concept. Learning surface: Experiment Tracking. What this page says: Experiment tracking records the config, code, data, metric curve, checkpoint, artifacts, and notes that make a run comparable instead of mysterious. Current section: Intuition, math, code, and interactive demo. Suggested next step: Manipulate one control and predict the visible change.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Explain the central idea in plain language, then restate it with the exact math objects from the page. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.

Domain

Production ML

production-mlmlopsreproducibilityexperiment-trackingevaluation

Prerequisites

Training Loss Curves and Debugging (review)Dataset Versioning (review)

Leads To

Evaluation Pipelines Cost and Latency Observability

Model Selection and Hyperparameter Search Practical Training Loop (review)Tiny LM Training Loop (review)

Within this domain