Continuous Function

NLP & Speech

Language-Model Evaluation and Perplexity

Score a held-out token sequence by average negative log-likelihood, then see why perplexity comparisons require the same data, tokenizer, and context window.

status: reviewimportance: importantdifficulty 3/5math: undergraduateread: 15mlive demo
Back to NLP & SpeechNext: Tiny LM Training Loop (review)

Concept Structure

Language-Model Evaluation and Perplexity

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

3prerequisites
3next concepts
3related links

Learner Contract

What this page should let you do.

You are here becauseScore a held-out token sequence by average negative log-likelihood, then see why perplexity comparisons require the same data, tokenizer, and context window.

This NLP & Speech concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Then go nextTiny LM Training Loop (review)

Follow this edge after making one prediction here; the next page should reuse the result, not restart the route.

Test the linkManipulate one control and predict the visible change.Then continue to Tiny LM Training Loop (review)

Claim/source review status

Substantive review recorded

1/1 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.
Claims1/1 reviewed
Sources4 cited
Codeattached
Demolive
Reviewed2026-07-03
Updatedpage 2026-07-03

Learning item flow

4/4 sections readyAsk about thisResearch room
ConceptLanguage-Model Evaluation and PerplexityNLP & SpeechEquationLanguage-Model Evaluation and Perplexity equation 1Exact equationCodeLanguage-Model Evaluation and Perplexity runnable code 1Exact runnable codeDemoLanguage-Model Evaluation and Perplexity interactive demoInteractive demoClaimPerplexity for an autoregressive language model is the exponential of...Exact claim checkSourceSpeech and Language Processing, Chapter 3: N-gram Language ModelsExact source
ConceptLanguage-Model Evaluation and PerplexityNLP & Speech
4 sources attachedLocal snapshot ready
concept:nlp-speech/language-model-evaluation-perplexity
CodenearbyPredictbefore revealRoomitem handoff
01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.
01

01

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

A language model can assign a probability to each next token in a held-out sequence. Evaluation asks a simple question:

How surprised was the model, on average, by the correct next token?

Raw sequence probability is not enough. Multiplying probabilities makes longer sequences look smaller even when the model is doing well at each step. Perplexity fixes the length problem by averaging the negative log-probability per token, then exponentiating that average.

That makes perplexity feel like an "effective choice count." A perplexity near 2 means the model is about as uncertain as choosing between two equally likely options at each token. A lower perplexity means the model assigned more probability to the actual held-out tokens.

But the comparison is only fair when the setup is aligned:

  • same held-out text
  • same tokenizer and vocabulary
  • same rule for context windows
  • same treatment of ignored tokens, unknown tokens, and sentence boundaries

Perplexity is useful, but it is not the whole evaluation story. A model can have good likelihood and still generate repetitive or unhelpful text under a bad decoding rule.

Source spine: Jurafsky and Martin, SLP3 Chapter 3; Stanford CS224N language-model notes; Hugging Face Transformers, Perplexity of fixed-length models; Holtzman et al., The Curious Case of Neural Text Degeneration.

02

02

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt
Equation 1x1:N=(x1,x2,…,xN).x_{1:N} = (x_1, x_2, \dots, x_N).Equation 2pθ(x1:N)=∏i=1Npθ(xi∣x<i).p_\theta(x_{1:N}) = \prod_{i=1}^{N} p_\theta(x_i \mid x_{<i}).

Let a held-out token sequence be

x1:N=(x1,x2,…,xN).x_{1:N} = (x_1, x_2, \dots, x_N).

An autoregressive language model factorizes the sequence probability into next-token probabilities:

pθ(x1:N)=∏i=1Npθ(xi∣x<i).p_\theta(x_{1:N}) = \prod_{i=1}^{N} p_\theta(x_i \mid x_{<i}).

The log-likelihood is a sum:

log⁡pθ(x1:N)=∑i=1Nlog⁡pθ(xi∣x<i).\log p_\theta(x_{1:N}) = \sum_{i=1}^{N} \log p_\theta(x_i \mid x_{<i}).

The average negative log-likelihood is:

NLL⁡avg=−1N∑i=1Nlog⁡pθ(xi∣x<i).\operatorname{NLL}_{\text{avg}} = -\frac{1}{N}\sum_{i=1}^{N}\log p_\theta(x_i \mid x_{<i}).

For a one-hot empirical target distribution over the correct next token, this average NLL is the token-level cross-entropy on the held-out sequence.

Perplexity with natural logarithms is:

PPL⁡(x1:N)=exp⁡(NLL⁡avg).\operatorname{PPL}(x_{1:N}) = \exp(\operatorname{NLL}_{\text{avg}}).

Equivalently:

PPL⁡(x1:N)=(∏i=1N1pθ(xi∣x<i))1/N.\operatorname{PPL}(x_{1:N}) = \left(\prod_{i=1}^{N}\frac{1}{p_\theta(x_i \mid x_{<i})}\right)^{1/N}.

That last form shows why one very low probability can hurt the score: perplexity is a geometric mean of inverse probabilities.

For fixed-context Transformer language models, the exact conditioning context may be truncated. A fair implementation should specify whether it uses disjoint chunks, a sliding window, or a strided sliding window, because those choices change which context the model sees before each scored token.

03

03

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt
Runnable code 1from math import exp, log tokens = ["the", "tiny", "model", "predicts", "</s>"] scores = { "s...python

This small script computes held-out average NLL and perplexity for two score profiles.

from math import exp, log

tokens = ["the", "tiny", "model", "predicts", "</s>"]
scores = {
    "steady": [0.70, 0.60, 0.50, 0.40, 0.60],
    "spiky": [0.90, 0.90, 0.10, 0.90, 0.10],
}

def evaluate(probs):
    nll = [-log(p) for p in probs]
    mean_nll = sum(nll) / len(nll)
    raw_probability = exp(sum(log(p) for p in probs))
    return mean_nll, exp(mean_nll), raw_probability, nll

for name, probs in scores.items():
    mean_nll, ppl, raw, nll = evaluate(probs)
    print(name)
    print("  token nll:", [round(v, 3) for v in nll])
    print("  mean nll:", round(mean_nll, 3))
    print("  perplexity:", round(ppl, 2))
    print("  raw probability:", f"{raw:.5f}")

The steady model wins on perplexity here because it avoids catastrophic misses, even though the spiky model is more confident on several tokens.

04

04

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Prediction check: inspect the held-out token tape and the qualitative scorecards, then predict which model has lower perplexity and which setup change would make the comparison unfair before revealing the log scores.

The lab hides the token probabilities, negative log-likelihood bars, mean NLL, raw probability, and final perplexity until you commit. The goal is to feel why perplexity is an average log score, not a raw sequence probability or a universal generation-quality grade.

Live Concept Demo

Explore Language-Model Evaluation and Perplexity

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 3/5undergraduatecode-aligned
Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what Language-Model Evaluation and Perplexity should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

Score a held-out token sequence by average negative log-likelihood, then see why perplexity comparisons require the same data, tokenizer, and context window.

Prediction open01 / Intuition
Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Language-Model Evaluation and Perplexity should make visible.

Visual Inquiry

Make the image answer a mathematical question

Score a held-out token sequence by average negative log-likelihood, then see why perplexity comparisons require the same data, tokenizer, and context window.

4/4 stages readyLive demo connected
Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Language-Model Evaluation and Perplexity easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

book · 2026Speech and Language Processing, Chapter 3: N-gram Language ModelsJurafsky and Martin

Source for held-out train/dev/test evaluation, sequence probability length dependence, and perplexity as a per-word geometric mean of inverse probabilities.

Open source
course-notes · 2019CS224N Lecture Notes Part V: Language Models, RNN, GRU and LSTMStanford CS224N

Course source for language models as sequence-probability models, corpus-level cross-entropy, and perplexity as a lower-is-better confidence measure.

Open source
documentation · 2026Transformers: Perplexity of fixed-length modelsHugging Face

Implementation-facing source for perplexity as exponentiated average negative log-likelihood, tokenizer dependence, and fixed-context sliding-window evaluation caveats.

Open source
paper · 2019The Curious Case of Neural Text DegenerationHoltzman, Buys, Du, Forbes, and Choi

Source for the caveat that likelihood-trained models and likelihood/decoding objectives do not by themselves guarantee good open-ended generated text.

Open source

Claim Review

Score a held-out token sequence by average negative log-likelihood, then see why perplexity comparisons require the same data, tokenizer, and context window.

Status1 substantive review recorded

Claims without a substantive review badge still need exact source-support review.

Sources4 references

jurafsky-martin-slp3-ngram-perplexity, cs224n-2019-lm-rnn-notes, hf-transformers-perplexity, holtzman-2019-neural-text-degeneration

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedPerplexity for an autoregressive language model is the exponential of average held-out negative log-likelihood, equivalently the exponential of token-level cross-entropy; it supports comparison only when test data, tokenization, and context-window treatment are aligned.Claim metadata: source checked

SLP3 supports held-out evaluation, raw probability length dependence, and perplexity as a normalized geometric mean. CS224N supports corpus cross-entropy and the lower-perplexity intuition. Hugging Face supports the modern causal-LM formula, tokenizer dependence, and fixed-context sliding-window caveat. Holtzman et al. support keeping generation-quality claims separate from likelihood/perplexity alone.

Sources: Speech and Language Processing, Chapter 3: N-gram Language Models, CS224N Lecture Notes Part V: Language Models, RNN, GRU and LSTM, Transformers: Perplexity of fixed-length models, The Curious Case of Neural Text DegenerationFive-token synthetic score table only; not a neural benchmark, confidence interval, or complete generation-quality evaluation.A bounded review summary is present; still check caveats and exact reference scope.

Checked SLP3 Chapter 3, CS224N LM/RNN notes, Hugging Face Transformers perplexity docs, and Holtzman et al. for held-out evaluation, cross-entropy/perplexity arithmetic, tokenizer and context-window caveats, and generation-quality limitations. GPT Pro/Oracle critique remains unavailable because 127.0.0.1:51672 is closed.

Reviewer: codex-local-primary-source-audit; reviewed 2026-07-03
source-span-jurafsky-martin-slp3-ngram-perplexitysource-span-cs224n-2019-lm-rnn-notessource-span-hf-transformers-perplexitysource-span-holtzman-2019-neural-text-degenerationEquation 1Equation 2Runnable code 1Interactive demo

Source support candidates

book 2026Speech and Language Processing, Chapter 3: N-gram Language Models

Source for held-out train/dev/test evaluation, sequence probability length dependence, and perplexity as a per-word geometric mean of inverse probabilities.

course-notes 2019CS224N Lecture Notes Part V: Language Models, RNN, GRU and LSTM

Course source for language models as sequence-probability models, corpus-level cross-entropy, and perplexity as a lower-is-better confidence measure.

documentation 2026Transformers: Perplexity of fixed-length models

Implementation-facing source for perplexity as exponentiated average negative log-likelihood, tokenizer dependence, and fixed-context sliding-window evaluation caveats.

paper 2019The Curious Case of Neural Text Degeneration

Source for the caveat that likelihood-trained models and likelihood/decoding objectives do not by themselves guarantee good open-ended generated text.

Mechanism checks

Equation 1
x1:N=(x1,x2,…,xN).x_{1:N} = (x_1, x_2, \dots, x_N).
Equation 2
pθ(x1:N)=∏i=1Npθ(xi∣x<i).p_\theta(x_{1:N}) = \prod_{i=1}^{N} p_\theta(x_i \mid x_{<i}).
Runnable code 1from math import exp, log tokens = ["the", "tiny", "model", "predicts", "</s>"] scores = { "s...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

Score a held-out token sequence by average negative log-likelihood, then see why perplexity comparisons require the same data, tokenizer, and context window.

Readiness0/3 checks ready
Predict

Before touching the demo, predict one visible change that should happen in Language-Model Evaluation and Perplexity.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Grounded research drawerClose
ConceptLanguage-Model Evaluation and PerplexityNLP & Speech
Runnable code comparisonLanguage-Model Evaluation and Perplexity runnable code 1scores = {Prediction before revealLanguage-Model Evaluation and Perplexity interactive demoManipulate one control and predict the visible change.
Grounded room questionWhat is the smallest example that makes Language-Model Evaluation and Perplexity click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.
Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptNLP & Speech

Language-Model Evaluation and Perplexity

Attached question

What is the smallest example that makes Language-Model Evaluation and Perplexity click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action
Local action draft

This draft stays in this browser, attached to the selected learning item.

No local draft saved.
Evidence to inspect
  • References to inspect: attached references on this page.
  • Definition, prerequisite, and contrast concept links
  • The equation or runnable code that makes the concept operational
  • One demo state that shows the invariant instead of a slogan
What would resolve this
  • The learner can state the mechanism in their own words
  • The learner can name the prerequisite that would repair confusion
  • The learner can predict how the mechanism changes under one perturbation
Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - Language-Model Evaluation and Perplexity Selected item key: recorded for copy. Context: NLP & Speech Page anchor: recorded for copy. Open question: What is the smallest example that makes Language-Model Evaluation and Perplexity click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context
concept/concept-notebook/nlp-speech/language-model-evaluation-perplexity concept:nlp-speech/language-model-evaluation-perplexity