Production ML

Dataset Cleaning, Filtering, and Deduplication

Data curation gates decide which documents become training distribution evidence, which duplicates amplify, and which benchmark-like examples must be removed before claims are trusted.

status: reviewimportance: criticaldifficulty 4/5math: graduateread: 18mlive demo

Back to Production ML Next: Evaluation Harnesses and Benchmark Contamination

Concept Structure

Dataset Cleaning, Filtering, and Deduplication

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

3prerequisites

2next concepts

3related links

Learner Contract

What this page should let you do.

You are here becauseData curation gates decide which documents become training distribution evidence, which duplicates amplify, and which benchmark-like examples must be removed before claims are trusted.

This Production ML concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

Before thisTrain/Dev/Test Splits, Cross-Validation, and Leakage

3 prerequisites listed; refresh them before leaning on the math or code.

Train/Dev/Test Splits, Cross-Validation, and Leakage Dataset Versioning (review)Pretraining Data Mixtures: Designing the Token Distribution (review)

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Then go nextEvaluation Harnesses and Benchmark Contamination

Follow this edge after making one prediction here; the next page should reuse the result, not restart the route.

Test the linkManipulate one control and predict the visible change.Then continue to Evaluation Harnesses and Benchmark Contamination

Claim/source review status

Substantive review recorded

1/1 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.

Claims1/1 reviewed

Sources6 cited

Codeattached

Demolive

Reviewed2026-07-03

Updatedpage 2026-07-03

Learning item flow

4/4 sections readyAsk about this Research room

ConceptDataset Cleaning, Filtering, and DeduplicationProduction ML EquationDataset Cleaning, Filtering, and Deduplication equation 1Exact equation CodeDataset Cleaning, Filtering, and Deduplication runnable code 1Exact runnable code DemoDataset Cleaning, Filtering, and Deduplication interactive demoInteractive demo ClaimCleaning, filtering, deduplication, PII policy, and benchmark-overlap...Exact claim check SourceStanford CS336: Language Modeling from ScratchExact source

ConceptDataset Cleaning, Filtering, and DeduplicationProduction ML

6 sources attachedLocal snapshot ready

concept:production-ml/dataset-cleaning-filtering-deduplication

Codenearby Predictbefore reveal Roomitem handoff

01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

A pretraining dataset is not raw text poured into a model. It is the result of gates. Some documents fail quality filters, some are near-duplicates of better documents, some contain private or unsafe material, and some look too much like evaluation data.

That means “we trained on $D$ tokens” is incomplete. The sharper question is:

Which documents survived the curation gates, and what distribution did those survivors create?

For serious LLM work, cleaning and deduplication are not housekeeping. They change which domains are sampled, how often repeated templates appear, whether private or benchmark-like text can leak into training, and whether a later evaluation score is interpretable.

This page sits between dataset versioning and pretraining data mixtures. Dataset versioning asks whether the evidence bundle changed. Data mixtures ask how retained sources are weighted. Dataset curation asks which raw documents are allowed to become retained evidence in the first place.

Source spine: Stanford CS336 motivates data and training as implementation objects in a language-model-from-scratch route. The FineWeb and FineWeb-Edu dataset cards expose filtering, deduplication, and ablation tradeoffs in public web data. Dolma shows open-corpus curation with quality, PII, deduplication, and contamination checks. DataComp-LM treats data curation as an evaluated experimental variable. Lee et al., “Deduplicating Training Data Makes Language Models Better”, is a useful primary source for deduplication and memorization evidence.

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt

Equation 1 $g_j = q_j(\tau)\,p_j(\pi)\,b_j(\lambda)\,d_j(\delta).$ Equation 2 $\mathcal D_{\mathrm{keep}}=\{x_j:g_j=1\}, \qquad N_{\mathrm{keep}}=\sum_{j:g_j=1} n_j.$

Let the raw corpus be documents $x_j$ with metadata $m_j$ , token count $n_j$ , quality score $s_j$ , and source label $a_j$ . A curation policy has four visible gates:

g_j = q_j(\tau)\,p_j(\pi)\,b_j(\lambda)\,d_j(\delta).

Here:

$q_j(\tau)=\mathbf 1\{s_j\ge \tau\}$ is a quality threshold.
$p_j(\pi)$ is the privacy or policy gate, such as keep, redact, or remove.
$b_j(\lambda)$ is a benchmark-overlap gate under similarity threshold $\lambda$ .
$d_j(\delta)$ is the deduplication retention decision under near-duplicate strictness $\delta$ .

The retained corpus is

\mathcal D_{\mathrm{keep}}=\{x_j:g_j=1\}, \qquad N_{\mathrm{keep}}=\sum_{j:g_j=1} n_j.

Deduplication is usually a corpus-level operation, not an independent document test. One simple abstraction builds a graph $G_\delta$ whose nodes are documents and whose edges connect pairs with similarity at least $\delta$ . For each connected component $C$ , retain a representative such as

j^\star(C)=\arg\max_{j\in C} u_j,

where $u_j$ may combine source priority, quality score, recency, licensing, and provenance. The toy demo uses quality score as $u_j$ so the decision is visible; real systems need more careful policy.

The retained token distribution is

Q_{\mathrm{keep}}(a) = \frac{\sum_{j:g_j=1,\ a_j=a} n_j}{N_{\mathrm{keep}}}.

That distribution can differ sharply from the raw distribution:

\operatorname{TV}(Q_{\mathrm{raw}},Q_{\mathrm{keep}}) = \frac12\sum_a |Q_{\mathrm{raw}}(a)-Q_{\mathrm{keep}}(a)|.

A contamination diagnostic can be written as

\kappa = \frac{\sum_j n_j\,\mathbf 1\{\operatorname{sim}(x_j,E)\ge \lambda\}\,g_j}{\sum_j n_jg_j}.

The ideal is not “remove everything suspicious until the corpus is tiny.” The ideal is an inspectable curation contract: what was filtered, what was deduplicated, what was retained, what shifted, and what claims remain unsupported.

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt

Runnable code 1docs = [ ("01", "web", .92, "quantum", 1200, False, False), ("02", "web", .90, "quantum", 115...python

This witness makes the gates explicit on a tiny corpus. It is not a legal, safety, or production data-governance system.

docs = [
    ("01", "web", .92, "quantum", 1200, False, False),
    ("02", "web", .90, "quantum", 1150, False, True),
    ("03", "edu", .78, "linear", 1400, False, False),
    ("04", "edu", .72, "linear", 1380, False, False),
    ("05", "web", .44, "lookup", 700, True, False),
    ("06", "code", .86, "loops", 1000, False, False),
    ("07", "code", .58, "loops", 950, False, False),
    ("08", "web", .69, "eval", 900, False, True),
    ("09", "web", .61, "cooking", 800, False, False),
    ("10", "edu", .88, "proofs", 1000, False, False),
]
sims = {("01", "02"): .96, ("03", "04"): .91, ("06", "07"): .84}

def curate(threshold=.60, dedup_threshold=.90, remove_pii=True, remove_bench=True):
    kept, reasons = {}, {}
    for doc_id, src, quality, group, tokens, pii, bench in docs:
        reason = []
        if quality < threshold: reason.append("low_quality")
        if remove_pii and pii: reason.append("pii")
        if remove_bench and bench: reason.append("benchmark_overlap")
        if reason: reasons[doc_id] = reason
        else: kept[doc_id] = (src, quality, group, tokens)

    for (a, b), sim in sims.items():
        if sim >= dedup_threshold and a in kept and b in kept:
            loser = min((a, b), key=lambda z: kept[z][1])
            reasons[loser] = ["near_duplicate"]
            kept.pop(loser)

    token_totals = {}
    for src, _, _, tokens in kept.values():
        token_totals[src] = token_totals.get(src, 0) + tokens
    return sorted(kept), reasons, token_totals

print(curate())

The default policy keeps documents 01, 03, 06, 09, and 10. Document 04 is not “bad text”; it is a near-duplicate of a stronger retained document. Document 08 may be good prose, but the benchmark-overlap gate removes it because evaluation trust matters.

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Prediction check: choose a curation policy, then predict which candidate document survives all gates and which removed-risk category dominates before revealing the ledger.

The widget hides the kept-document set, removed-token totals, duplicate-cluster winner, retained source distribution, and contamination mass until you commit. When you reveal, ask the audit question: did the dataset get cleaner, or did the retained distribution also move in a way the training run must report?

Live Concept Demo

Explore Dataset Cleaning, Filtering, and Deduplication

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 4/5graduatecode-aligned

Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what Dataset Cleaning, Filtering, and Deduplication should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

Data curation gates decide which documents become training distribution evidence, which duplicates amplify, and which benchmark-like examples must be removed before claims are trusted.

Prediction open01 / Intuition

Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Dataset Cleaning, Filtering, and Deduplication should make visible.

Visual Inquiry

Make the image answer a mathematical question

Data curation gates decide which documents become training distribution evidence, which duplicates amplify, and which benchmark-like examples must be removed before claims are trusted.

4/4 stages readyLive demo connected

Visual cueWhich visible object should carry the first intuition?

Inspection depth2/4

Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Dataset Cleaning, Filtering, and Deduplication easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

course-notes · 2026Stanford CS336: Language Modeling from ScratchStanford CS336 course staff

Course-level source for treating data collection, tokenization, training, and evaluation as implementation objects in a language-model-from-scratch route.

Open source

documentation · 2024FineWeb: decanting the web for the finest text data at scaleHugging Face

Public dataset card exposing crawl selection, filtering, deduplication, and ablation-driven curation choices for web-scale pretraining data.

Open source

documentation · 2024FineWeb-EduHugging Face

Public dataset card for educational-quality filtering thresholds, including the quality-retention tradeoff and benchmark-style ablation framing.

Open source

paper · 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining ResearchSoldaini et al.

Supports multi-source corpus construction, quality filters, PII handling, deduplication, and contamination-oriented checks in an open pretraining corpus.

Open source

paper · 2024DataComp-LM: In search of the next generation of training sets for language modelsLi et al.

Supports treating data curation as an experimental variable evaluated under controlled training and benchmark protocols.

Open source

paper · 2022Deduplicating Training Data Makes Language Models BetterLee et al.

Primary source for deduplication reducing memorization and improving downstream behavior in studied language-model settings, with caveats about method and corpus scope.

Open source

Claim Review

Data curation gates decide which documents become training distribution evidence, which duplicates amplify, and which benchmark-like examples must be removed before claims are trusted.

Status1 substantive review recorded

Claims without a substantive review badge still need exact source-support review.

Sources6 references

cs336-language-modeling-from-scratch, fineweb-dataset-card, fineweb-edu-dataset-card, dolma-open-corpus, datacomp-lm, deduplicating-training-data

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedCleaning, filtering, deduplication, PII policy, and benchmark-overlap checks are not cosmetic preprocessing steps: they define the retained training distribution, repetition pressure, and evaluation-contamination risk that a model update can see.Claim metadata: source checked

The sources jointly support modeling data curation as a retained-corpus contract with explicit filtering, deduplication, contamination, and evaluation consequences rather than an invisible preprocessing footnote.

Sources: Stanford CS336: Language Modeling from Scratch, FineWeb: decanting the web for the finest text data at scale, FineWeb-Edu, Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research, DataComp-LM: In search of the next generation of training sets for language models, Deduplicating Training Data Makes Language Models BetterThe interactive demo uses a tiny deterministic corpus and simple thresholds. It is not a claim about optimal real-world thresholds, legal/PII compliance, licensing, safety filtering, production governance, or benchmark validity without source-specific audits.A bounded review summary is present; still check caveats and exact reference scope.

Checked Stanford CS336 as route-level language-model-from-scratch motivation, FineWeb/FineWeb-Edu dataset cards for public filtering/dedup/ablation tradeoffs, Dolma for open corpus curation and risk handling, DataComp-LM for curation-as-evaluated-variable framing, and Lee et al. for deduplication/memorization evidence. GPT Pro/Oracle publication critique is still pending because 127.0.0.1:51672 is unavailable.

Reviewer: codex-local-primary-source-audit; reviewed 2026-07-03

source-span-cs336-language-modeling-from-scratch source-span-fineweb-dataset-card source-span-dolma-open-corpus source-span-deduplicating-training-data Equation 1 Runnable code 1 Interactive demo

Source support candidates

course-notes 2026Stanford CS336: Language Modeling from Scratch

Course-level source for treating data collection, tokenization, training, and evaluation as implementation objects in a language-model-from-scratch route.

documentation 2024FineWeb: decanting the web for the finest text data at scale

Public dataset card exposing crawl selection, filtering, deduplication, and ablation-driven curation choices for web-scale pretraining data.

documentation 2024FineWeb-Edu

Public dataset card for educational-quality filtering thresholds, including the quality-retention tradeoff and benchmark-style ablation framing.

paper 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research

Supports multi-source corpus construction, quality filters, PII handling, deduplication, and contamination-oriented checks in an open pretraining corpus.

Mechanism checks

Equation 1

g_j = q_j(\tau)\,p_j(\pi)\,b_j(\lambda)\,d_j(\delta).

Equation 2

\mathcal D_{\mathrm{keep}}=\{x_j:g_j=1\}, \qquad N_{\mathrm{keep}}=\sum_{j:g_j=1} n_j.

Runnable code 1docs = [ ("01", "web", .92, "quantum", 1200, False, False), ("02", "web", .90, "quantum", 115...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

Data curation gates decide which documents become training distribution evidence, which duplicates amplify, and which benchmark-like examples must be removed before claims are trusted.

Readiness0/3 checks ready

Predict

Before touching the demo, predict one visible change that should happen in Dataset Cleaning, Filtering, and Deduplication.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Your answer canvas

Local checks

Claim

A concrete answer is on the canvas.

Mechanism

The answer names why the claim should hold.

Bridge

It touches the page context or a neighboring idea.

Misconception check

Grounded research drawerClose

ConceptDataset Cleaning, Filtering, and DeduplicationProduction ML

Runnable code comparisonDataset Cleaning, Filtering, and Deduplication runnable code 1docs = [ ("01", "web", .92, "quantum", 1200, False, False), ("02", "web", .90, "quantum", 115...Prediction before revealDataset Cleaning, Filtering, and Deduplication interactive demoManipulate one control and predict the visible change.

Grounded room questionWhat is the smallest example that makes Dataset Cleaning, Filtering, and Deduplication click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptProduction ML

Dataset Cleaning, Filtering, and Deduplication

Attached question

What is the smallest example that makes Dataset Cleaning, Filtering, and Deduplication click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action

Local action draft

This draft stays in this browser, attached to the selected learning item.

Draft noteNext action

No local draft saved.

Evidence to inspect

References to inspect: attached references on this page.
Definition, prerequisite, and contrast concept links
The equation or runnable code that makes the concept operational
One demo state that shows the invariant instead of a slogan

What would resolve this

The learner can state the mechanism in their own words
The learner can name the prerequisite that would repair confusion
The learner can predict how the mechanism changes under one perturbation

Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - Dataset Cleaning, Filtering, and Deduplication Selected item key: recorded for copy. Context: Production ML Page anchor: recorded for copy. Open question: What is the smallest example that makes Dataset Cleaning, Filtering, and Deduplication click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context

concept/concept-notebook/production-ml/dataset-cleaning-filtering-deduplication
concept:production-ml/dataset-cleaning-filtering-deduplication

Learning Map

Before / Now / Try / Next

BeforeTrain/Dev/Test Splits, Cross-Validation, and Leakage

NowIntuition -> Math -> Code -> Demo

TryManipulate one control and predict the visible change.

NextEvaluation Harnesses and Benchmark Contamination

Intuitionready
Mathready
Codeready
Interactive Demoready

AI Companion

Ask beside the selected object

Data curation gates decide which documents become training distribution evidence, which duplicates amplify, and which benchmark-like examples must be removed before claims are trusted.

Your question

GoalComfortStyleStuck on

Context prompt

You are my AI learning companion for Continuous Function. Current context: Production ML concept. Learning surface: Dataset Cleaning, Filtering, and Deduplication. What this page says: Data curation gates decide which documents become training distribution evidence, which duplicates amplify, and which benchmark-like examples must be removed before claims are trusted. Current section: Intuition, math, code, and interactive demo. Suggested next step: Manipulate one control and predict the visible change.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Explain the central idea in plain language, then restate it with the exact math objects from the page. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.

Domain

Production ML

production-mlllm-from-scratchdata-curationdeduplicationevaluation

Prerequisites

Train/Dev/Test Splits, Cross-Validation, and Leakage Dataset Versioning (review)Pretraining Data Mixtures: Designing the Token Distribution (review)

Leads To

Evaluation Harnesses and Benchmark Contamination Raw Text to Packed Causal-LM Examples (review)

Training Loss Curves and Debugging (review)Scaling Laws & Emergent Abilities Tokenization & Vocabulary Design

Within this domain