Continuous Function

Attention & Transformers

Decoder-Only Transformer Forward Pass

A shape-first walk from token IDs through a decoder-only Transformer block to logits and masked next-token loss.

status: reviewimportance: criticaldifficulty 4/5math: undergraduateread: 14mlive demo
Back to Attention & TransformersNext: Tiny LM Training Loop (review)

Concept Structure

Decoder-Only Transformer Forward Pass

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

10prerequisites
4next concepts
3related links

Learner Contract

What this page should let you do.

You are here becauseA shape-first walk from token IDs through a decoder-only Transformer block to logits and masked next-token loss.

This Attention & Transformers concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Then go nextTiny LM Training Loop (review)

Follow this edge after making one prediction here; the next page should reuse the result, not restart the route.

Test the linkManipulate one control and predict the visible change.Then continue to Tiny LM Training Loop (review)

Claim/source review status

Substantive review recorded

1/1 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.
Claims1/1 reviewed
Sources3 cited
Codeattached
Demolive
Reviewed2026-07-02
Updatedpage 2026-07-02

Learning item flow

4/4 sections readyAsk about thisResearch room
ConceptDecoder-Only Transformer Forward PassAttention & TransformersEquationDecoder-Only Transformer Forward Pass equation 1Exact equationCodeDecoder-Only Transformer Forward Pass runnable code 1Exact runnable codeDemoDecoder-Only Transformer Forward Pass interactive demoInteractive demoClaimA minimal decoder-only Transformer language-model forward pass maps t...Exact claim checkSourceStanford CS336 Assignment 1: BasicsExact source
ConceptDecoder-Only Transformer Forward PassAttention & Transformers
4 sources attachedLocal snapshot ready
concept:attention-transformers/decoder-only-transformer-forward-pass
CodenearbyPredictbefore revealRoomitem handoff
01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.
01

01

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

The first useful Transformer-from-scratch question is not "how do I train GPT?" It is smaller and sharper:

If the input is a batch of token IDs, where does the vocabulary dimension first appear?

Inside the block, the model mostly preserves a hidden-state shape. Token IDs start as a table of integers shaped (B,T), where B is batch size and T is sequence length. Embedding turns each integer into a vector, so the residual stream becomes (B,T,d_model). Causal self-attention returns another (B,T,d_model) object. The MLP branch returns (B,T,d_model) too. That sameness is what makes residual addition legal.

Only the final output projection changes the last dimension from d_model to the vocabulary size V. Those logits are then paired with shifted next-token labels and a loss mask from the training example.

This page is a local shape probe, not a training-quality implementation. It is meant to make the forward pass inspectable before we add stacking, optimizers, checkpoints, data scale, or performance kernels.

02

02

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt
Equation 1H0[b,t,:]=Etok[X[b,t]]+Epos[t],H_0[b,t,:] = E_{\text{tok}}[X[b,t]] + E_{\text{pos}}[t],Equation 2H~0=LN⁡(H0),\tilde{H}_0 = \operatorname{LN}(H_0),

Let:

  • B be batch size.
  • T be sequence length.
  • d be d_model, the hidden width.
  • V be vocabulary size.
  • X in {0,...,V-1}^{B x T} be token IDs.
  • E_tok in R^{V x d} be the token embedding table.
  • E_pos in R^{T x d} be a learned positional embedding table for this local shape probe.

The input hidden state is

H0[b,t,:]=Etok[X[b,t]]+Epos[t],H_0[b,t,:] = E_{\text{tok}}[X[b,t]] + E_{\text{pos}}[t],

so H_0 has shape (B,T,d).

For a single pre-norm decoder block, the attention branch is applied to normalized hidden states:

H~0=LN⁡(H0),\tilde{H}_0 = \operatorname{LN}(H_0), Q=H~0WQ,K=H~0WK,R=H~0WV.Q = \tilde{H}_0 W_Q,\quad K = \tilde{H}_0 W_K,\quad R = \tilde{H}_0 W_V.

For query position i and key position j, the causal mask is

Mij={0j≤i−∞j>i.M_{ij} = \begin{cases} 0 & j \le i \\ -\infty & j > i. \end{cases}

The toy single-head attention output is

A=softmax⁡(QK⊤d+M)RWO.A = \operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d}} + M\right)R W_O.

A has shape (B,T,d), so the first residual update is legal:

H1=H0+A.H_1 = H_0 + A.

The MLP branch also returns the residual-stream width:

F=ϕ(LN⁡(H1)W1)W2,H2=H1+F.F = \phi(\operatorname{LN}(H_1)W_1)W_2,\quad H_2 = H_1 + F.

The vocabulary dimension appears when we unembed the final hidden states:

Z=LN⁡(H2)WU+bU,Z = \operatorname{LN}(H_2) W_U + b_U,

where W_U in R^{d x V} and Z has shape (B,T,V).

For causal language modeling, logit row Z[b,t,:] predicts the next label Y[b,t] = X[b,t+1], except positions excluded by the loss mask. With active positions \Omega, the masked cross-entropy is

L=1∣Ω∣∑(b,t)∈Ω−log⁡exp⁡Z[b,t,Y[b,t]]∑v=1Vexp⁡Z[b,t,v].\mathcal{L} = \frac{1}{|\Omega|}\sum_{(b,t)\in\Omega} -\log \frac{\exp Z[b,t,Y[b,t]]}{\sum_{v=1}^{V}\exp Z[b,t,v]}.
03

03

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt
Runnable code 1import numpy as np rng = np.random.default_rng(0) B, T, d, V = 1, 6, 4, 8 ids = np.array([[1,...python
import numpy as np

rng = np.random.default_rng(0)
B, T, d, V = 1, 6, 4, 8
ids = np.array([[1, 3, 4, 2, 1, 5]])
labels = np.array([[3, 4, 2, 1, -100, -100]])
active = labels != -100

tok = rng.normal(size=(V, d)) * 0.2
pos = rng.normal(size=(T, d)) * 0.2
Wq = rng.normal(size=(d, d)) * 0.2
Wk = rng.normal(size=(d, d)) * 0.2
Wv = rng.normal(size=(d, d)) * 0.2
Wo = rng.normal(size=(d, d)) * 0.2
W1 = rng.normal(size=(d, 2 * d)) * 0.2
W2 = rng.normal(size=(2 * d, d)) * 0.2
Wu = rng.normal(size=(d, V)) * 0.2

def ln(x, eps=1e-5):
    return (x - x.mean(-1, keepdims=True)) / np.sqrt(x.var(-1, keepdims=True) + eps)

def softmax(x):
    e = np.exp(x - np.max(x, axis=-1, keepdims=True))
    return e / e.sum(axis=-1, keepdims=True)

h = tok[ids] + pos[None, :, :]
q, k, v = ln(h) @ Wq, ln(h) @ Wk, ln(h) @ Wv
mask = np.triu(np.full((T, T), -np.inf), k=1)
attn = softmax(q @ np.swapaxes(k, -1, -2) / np.sqrt(d) + mask) @ v
h = h + attn @ Wo
h = h + np.tanh(ln(h) @ W1) @ W2
logits = ln(h) @ Wu
log_probs = logits - np.log(np.exp(logits).sum(axis=-1, keepdims=True))
rows = np.arange(active.sum())
loss = -log_probs[active][rows, labels[active]].mean()

print(h.shape, logits.shape, round(float(loss), 4))
04

04

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

The lab below hides the answer-bearing shapes until you commit.

Before reveal, predict which stage first changes the residual stream's last dimension from d_model to the vocabulary size V. After reveal, inspect the full path from token IDs to hidden states, logits, shifted labels, active loss positions, and masked mean cross-entropy.

The important invariant is simple: attention and the MLP update the residual stream; the output projection turns hidden states into vocabulary logits.

Live Concept Demo

Explore Decoder-Only Transformer Forward Pass

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 4/5undergraduatecode-aligned
Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what Decoder-Only Transformer Forward Pass should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

A shape-first walk from token IDs through a decoder-only Transformer block to logits and masked next-token loss.

Prediction open01 / Intuition
Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Decoder-Only Transformer Forward Pass should make visible.

Visual Inquiry

Make the image answer a mathematical question

A shape-first walk from token IDs through a decoder-only Transformer block to logits and masked next-token loss.

4/4 stages readyLive demo connected
Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Decoder-Only Transformer Forward Pass easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

course-notes · 2025Stanford CS336 Assignment 1: BasicsStanford CS336 course staff

Route-level source for implementing tokenizer, Transformer language-model architecture, optimizer, and a minimal language model from scratch. The public adapters expose the same-shape Transformer block contract and LM logits shape.

Open source
reference · 2026Dive into Deep Learning: TransformerZhang, Lipton, Li, and Smola

Pedagogical source for Transformer residual connections, layer normalization, positionwise feed-forward networks, and masked decoder self-attention.

Open source
paper · 2017Attention Is All You NeedVaswani et al.

Primary Transformer source for scaled dot-product attention, decoder masking, residual connections around sublayers, layer normalization, and linear-softmax output generation.

Open source
course-notes · 2023CS224N: Self-Attention and TransformersStanford CS224N course staff

NLP sequence-modeling source for next-token language-model framing and Transformer self-attention pedagogy.

Open source

Claim Review

A shape-first walk from token IDs through a decoder-only Transformer block to logits and masked next-token loss.

Status1 substantive review recorded

Claims without a substantive review badge still need exact source-support review.

Sources4 references

cs336-assignment1-basics, d2l-transformer-chapter, vaswani-2017-attention, cs224n-self-attention-transformers

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedA minimal decoder-only Transformer language-model forward pass maps token IDs shaped (B,T) to hidden states shaped (B,T,d_model) through same-shape attention and MLP residual updates, then applies a final vocabulary projection to produce logits shaped (B,T,V) for next-token loss.Claim metadata: source checked

CS336 supports the block and LM output shape contracts; D2L and Vaswani support residual, normalization, feed-forward, mask, and output-projection mechanics.

Sources: Stanford CS336 Assignment 1: Basics, Dive into Deep Learning: Transformer, Attention Is All You NeedThis page checks a one-block local shape probe. It does not claim benchmark quality, production training, exact PyTorch module parity, KV-cache behavior, weight tying, RoPE implementation details, or modern architecture variants such as grouped-query attention.A bounded review summary is present; still check caveats and exact reference scope.

Checked CS336 course/assignment pages and adapters, D2L Transformer text, Vaswani et al. Transformer sections, and the existing Continuous Function packed-example and causal-mask concepts. The page remains review-status pending GPT Pro publication critique.

Reviewer: codex-local-source-audit; reviewed 2026-07-02
source-span-cs336-assignment1-basicssource-span-d2l-transformer-chaptersource-span-vaswani-2017-attentionEquation 1Equation 2Runnable code 1Interactive demo

Source support candidates

course-notes 2025Stanford CS336 Assignment 1: Basics

Route-level source for implementing tokenizer, Transformer language-model architecture, optimizer, and a minimal language model from scratch. The public adapters expose the same-shape Transformer block contract and LM logits shape.

reference 2026Dive into Deep Learning: Transformer

Pedagogical source for Transformer residual connections, layer normalization, positionwise feed-forward networks, and masked decoder self-attention.

paper 2017Attention Is All You Need

Primary Transformer source for scaled dot-product attention, decoder masking, residual connections around sublayers, layer normalization, and linear-softmax output generation.

course-notes 2023CS224N: Self-Attention and Transformers

NLP sequence-modeling source for next-token language-model framing and Transformer self-attention pedagogy.

Mechanism checks

Equation 1
H0[b,t,:]=Etok[X[b,t]]+Epos[t],H_0[b,t,:] = E_{\text{tok}}[X[b,t]] + E_{\text{pos}}[t],
Equation 2
H~0=LN⁡(H0),\tilde{H}_0 = \operatorname{LN}(H_0),
Runnable code 1import numpy as np rng = np.random.default_rng(0) B, T, d, V = 1, 6, 4, 8 ids = np.array([[1,...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

A shape-first walk from token IDs through a decoder-only Transformer block to logits and masked next-token loss.

Readiness0/3 checks ready
Predict

Before touching the demo, predict one visible change that should happen in Decoder-Only Transformer Forward Pass.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Grounded research drawerClose
ConceptDecoder-Only Transformer Forward PassAttention & Transformers
Runnable code comparisonDecoder-Only Transformer Forward Pass runnable code 1attn = softmax(q @ np.swapaxes(k, -1, -2) / np.sqrt(d) + mask) @ vPrediction before revealDecoder-Only Transformer Forward Pass interactive demoManipulate one control and predict the visible change.
Grounded room questionWhat is the smallest example that makes Decoder-Only Transformer Forward Pass click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.
Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptAttention & Transformers

Decoder-Only Transformer Forward Pass

Attached question

What is the smallest example that makes Decoder-Only Transformer Forward Pass click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action
Local action draft

This draft stays in this browser, attached to the selected learning item.

No local draft saved.
Evidence to inspect
  • References to inspect: attached references on this page.
  • Definition, prerequisite, and contrast concept links
  • The equation or runnable code that makes the concept operational
  • One demo state that shows the invariant instead of a slogan
What would resolve this
  • The learner can state the mechanism in their own words
  • The learner can name the prerequisite that would repair confusion
  • The learner can predict how the mechanism changes under one perturbation
Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - Decoder-Only Transformer Forward Pass Selected item key: recorded for copy. Context: Attention & Transformers Page anchor: recorded for copy. Open question: What is the smallest example that makes Decoder-Only Transformer Forward Pass click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context
concept/concept-notebook/attention-transformers/decoder-only-transformer-forward-pass concept:attention-transformers/decoder-only-transformer-forward-pass