Continuous Function

Attention & Transformers

SwiGLU: Gated MLP Blocks in Transformers

Why modern transformer MLPs often use a learned multiplicative gate: one projection proposes a token-local write while another projection controls how much of each channel reaches the residual stream.

status: reviewimportance: importantdifficulty 4/5math: graduateread: 16mlive demo
Back to Attention & TransformersNext: Sparse Mixture of Experts: Routing, Load Balancing & Expert Parallelism

Concept Structure

SwiGLU: Gated MLP Blocks in Transformers

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

2prerequisites
1next concepts
3related links

Learning map

SwiGLU: Gated MLP Blocks in Transformers
BeforeScaled Dot-Product Attention & Transformer LayersNow4/4 sections readyTryManipulate one control and predict the visible change.NextSparse Mixture of Experts: Routing, Load Balancing & Expert Parallelism

Object flow

4/4 sections readyAsk about thisResearch room
ConceptSwiGLU: Gated MLP Blocks in TransformersAttention & TransformersEquationSwiGLU: Gated MLP Blocks in Transformers equation 1Exact equation objectCodeSwiGLU: Gated MLP Blocks in Transformers code witness 1Exact code witnessDemoSwiGLU: Gated MLP Blocks in Transformers interactive demoVisualization objectClaimSwiGLU: Gated MLP Blocks in Transformers central claimMechanism claimEquationSwiGLU: Gated MLP Blocks in Transformers math objectsEquation and notation
ConceptSwiGLU: Gated MLP Blocks in TransformersAttention & Transformers
Local snapshot ready
concept:attention-transformers/swiglu
Codewitness nearbyPredictbefore revealRoomobject handoff

Conceptual Bridge

What should feel connected as you move through this page.

Carry inScaled Dot-Product Attention & Transformer Layers

Bring the mental model from Scaled Dot-Product Attention & Transformer Layers; this page will reuse it instead of restarting from zero.

Work hereSwiGLU: Gated MLP Blocks in Transformers

Why modern transformer MLPs often use a learned multiplicative gate: one projection proposes a token-local write while another projection controls how much of each channel reaches the residual stream.

Carry outSparse Mixture of Experts: Routing, Load Balancing & Expert Parallelism

The next edge should feel earned: use the demo prediction here before following Sparse Mixture of Experts: Routing, Load Balancing & Expert Parallelism.

Test the linkManipulate one control and predict the visible change.Then continue to Sparse Mixture of Experts: Routing, Load Balancing & Expert Parallelism
01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.
01

01

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

Canonical sources: Shazeer, "GLU Variants Improve Transformer", Chowdhery et al., "PaLM: Scaling Language Modeling with Pathways", Touvron et al., "LLaMA: Open and Efficient Foundation Language Models", and Vaswani et al., "Attention Is All You Need".

Attention is the token-mixing part of a transformer block: each token reads from other tokens. The MLP or feedforward sublayer is the token-local writing part: each token takes its own residual vector, expands it into hidden channels, applies a nonlinearity, and writes a correction back.

A plain ReLU or GELU feedforward block asks each hidden channel one question: "how active is this feature?" SwiGLU asks two questions. One projection proposes a value. A second projection produces a gate. The final hidden channel is their product after passing the gate logit through SiLU.

That small change matters conceptually. The MLP is no longer just "linear, activation, linear." It becomes a bank of conditional writes: feature channels can be suppressed, passed, or amplified depending on another learned view of the same token state.

This page is only about the channel-level mechanism. It is not a benchmark claim that SwiGLU always beats GELU, and the common two-thirds hidden-width rule is a parameter-budget comparison, not a law of nature.

02

02

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt
Equation 1x \in \mathbb R^{d_{\text{model}}}Equation 22d_{\text{model}}d_{\text{ff}}

Use row-vector notation for one token. Let

x∈Rdmodelx \in \mathbb R^{d_{\text{model}}}

be the residual stream vector for that token. A standard two-matrix feedforward block with hidden width dffd_{\text{ff}} has roughly

2dmodeldff2d_{\text{model}}d_{\text{ff}}

matrix parameters, ignoring biases.

SwiGLU uses two input projections and one output projection. Choose a gated hidden width dff′d_{\text{ff}}^{\prime} and define

Wv,Wg∈Rdmodel×dff′,Wo∈Rdff′×dmodel.W_v, W_g \in \mathbb R^{d_{\text{model}}\times d_{\text{ff}}^{\prime}}, \qquad W_o \in \mathbb R^{d_{\text{ff}}^{\prime}\times d_{\text{model}}}.

The value and gate logits are

v=xWv∈Rdff′,g=xWg∈Rdff′.v=xW_v \in \mathbb R^{d_{\text{ff}}^{\prime}}, \qquad g=xW_g \in \mathbb R^{d_{\text{ff}}^{\prime}}.

SiLU, also called Swish, is

SiLU⁡(z)=zσ(z),σ(z)=11+e−z.\operatorname{SiLU}(z)=z\sigma(z), \qquad \sigma(z)=\frac{1}{1+e^{-z}}.

The gated hidden vector is the elementwise product

h=v⊙SiLU⁡(g).h=v\odot \operatorname{SiLU}(g).

The token-local MLP write is

y=hWo=(v⊙SiLU⁡(g))Wo=(xWv⊙SiLU⁡(xWg))Wo.y=hW_o =\bigl(v\odot \operatorname{SiLU}(g)\bigr)W_o =\bigl(xW_v\odot \operatorname{SiLU}(xW_g)\bigr)W_o.

The parameter count is approximately

3dmodeldff′.3d_{\text{model}}d_{\text{ff}}^{\prime}.

To compare it to a two-matrix block with expansion width dffd_{\text{ff}}, set

dff′≈23dff.d_{\text{ff}}^{\prime}\approx \frac{2}{3}d_{\text{ff}}.

Then

3dmodeldff′≈3dmodel(23dff)=2dmodeldff.3d_{\text{model}}d_{\text{ff}}^{\prime} \approx 3d_{\text{model}}\left(\frac{2}{3}d_{\text{ff}}\right) =2d_{\text{model}}d_{\text{ff}}.

So the usual comparison is not "SwiGLU has a free extra matrix." It is "SwiGLU spends a similar budget differently: two narrower input projections create a learned multiplicative gate."

03

03

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt
Code witness 1import numpy as np def silu(z): return z / (1.0 + np.exp(-z)) d_model = 6 d_ff = 12 d_ff_prim...python

This witness mirrors the math exactly. It uses one token vector, two input projections, a SiLU gate, an elementwise product, and an output projection. The final assertion checks the common two-thirds budget comparison.

import numpy as np

def silu(z):
    return z / (1.0 + np.exp(-z))

d_model = 6
d_ff = 12
d_ff_prime = round((2.0 / 3.0) * d_ff)

x = np.array([[0.7, -0.4, 0.2, 1.1, -0.3, 0.5]])  # shape: (1, d_model)

rng = np.random.default_rng(7)
W_v = rng.normal(0.0, 0.25, size=(d_model, d_ff_prime))
W_g = rng.normal(0.0, 0.25, size=(d_model, d_ff_prime))
W_o = rng.normal(0.0, 0.25, size=(d_ff_prime, d_model))

v = x @ W_v                      # shape: (1, d_ff_prime)
g = x @ W_g                      # shape: (1, d_ff_prime)
gate = silu(g)                   # shape: (1, d_ff_prime)
h = v * gate                     # elementwise product
y = h @ W_o                      # shape: (1, d_model)

relu_ffn_params = 2 * d_model * d_ff
swiglu_params = 3 * d_model * d_ff_prime
ratio = swiglu_params / relu_ffn_params

assert v.shape == (1, d_ff_prime)
assert g.shape == (1, d_ff_prime)
assert gate.shape == (1, d_ff_prime)
assert h.shape == (1, d_ff_prime)
assert y.shape == (1, d_model)
assert 0.95 <= ratio <= 1.05

channel = 3
print({
    "v_i": float(v[0, channel]),
    "g_i": float(g[0, channel]),
    "SiLU(g_i)": float(gate[0, channel]),
    "product": float(h[0, channel]),
    "budget_ratio": ratio,
})

The printed channel is the scalar version of the interactive demo. One number proposes a hidden-channel write, another number controls the gate, and their product decides what reaches the output projection.

04

04

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Use the Gated MLP Write demo to inspect one synthetic hidden channel. Before reveal, you can see the value projection viv_i and the gate logit gig_i, but not SiLU⁡(gi)\operatorname{SiLU}(g_i) or the product.

Predict whether the gate suppresses, passes, or amplifies the value. Then reveal the gate coefficient, the product viSiLU⁡(gi)v_i\operatorname{SiLU}(g_i), and a small toy selected-channel contribution through the output projection. The full MLP write would sum many such channel contributions; this demo isolates one. The point is not to memorize a curve; it is to feel why modern transformer MLPs can act like conditional feature writers.

Live Concept Demo

Explore SwiGLU: Gated MLP Blocks in Transformers

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 4/5graduatecode-aligned
Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what SwiGLU: Gated MLP Blocks in Transformers should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

Why modern transformer MLPs often use a learned multiplicative gate: one projection proposes a token-local write while another projection controls how much of each channel reaches the residual stream.

Prediction open01 / Intuition
Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change SwiGLU: Gated MLP Blocks in Transformers should make visible.

Visual Inquiry

Make the image answer a mathematical question

Why modern transformer MLPs often use a learned multiplicative gate: one projection proposes a token-local write while another projection controls how much of each channel reaches the residual stream.

4/4 stages readyLive demo connected
Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make SwiGLU: Gated MLP Blocks in Transformers easier to reason about before the page gives the answer.

Claim Review

Why modern transformer MLPs often use a learned multiplicative gate: one projection proposes a token-local write while another projection controls how much of each channel reaches the residual stream.

StatusSubstantive claim review pending

Source IDs and witness objects are attached for review; they are not proof by themselves.

SourcesNo references

Add source metadata before claiming support.

Witnesses4 local objects

Use equation, code, and demo objects to check whether the source support is operational.

Source support candidates

No structured source note is attached yet.

Mechanism witnesses

Equation 1
x∈Rdmodelx \in \mathbb R^{d_{\text{model}}}
Equation 2
2dmodeldff2d_{\text{model}}d_{\text{ff}}
Code witness 1import numpy as np def silu(z): return z / (1.0 + np.exp(-z)) d_model = 6 d_ff = 12 d_ff_prim...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

Why modern transformer MLPs often use a learned multiplicative gate: one projection proposes a token-local write while another projection controls how much of each channel reaches the residual stream.

Readiness0/3 checks ready
Predict

Before touching the demo, predict one visible change that should happen in SwiGLU: Gated MLP Blocks in Transformers.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Object research drawerClose
ConceptSwiGLU: Gated MLP Blocks in TransformersAttention & Transformers
Code witness comparisonSwiGLU: Gated MLP Blocks in Transformers code witness 1This witness mirrors the math exactly. It uses one token vector, two input projections, a SiLU gate, an elementwise product, and an output projecti...Prediction before revealSwiGLU: Gated MLP Blocks in Transformers interactive demoManipulate one control and predict the visible change.
Grounded room questionWhat is the smallest example that makes SwiGLU: Gated MLP Blocks in Transformers click without losing the math?Local snapshot ready

Research Room

Attach the question to an exact object

Pick the concept, equation, source, code witness, claim, misconception, or demo state before asking for help. The handoff stays grounded to that object.
Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptAttention & Transformers

SwiGLU: Gated MLP Blocks in Transformers

Anchored question

What is the smallest example that makes SwiGLU: Gated MLP Blocks in Transformers click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action
Local action draft

This draft stays locally in this browser for concept:attention-transformers/swiglu.

No local draft saved.
Evidence to inspect
  • Definition, prerequisite, and contrast concept links
  • The equation or code witness that makes the concept operational
  • One demo state that shows the invariant instead of a slogan
What would resolve this
  • The learner can state the mechanism in their own words
  • The learner can name the prerequisite that would repair confusion
  • The learner can predict how the mechanism changes under one perturbation
Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - SwiGLU: Gated MLP Blocks in Transformers Object key: concept:attention-transformers/swiglu Context: Attention & Transformers Anchor id: concept/concept-notebook/attention-transformers/swiglu Open question: What is the smallest example that makes SwiGLU: Gated MLP Blocks in Transformers click without losing the math? Evidence to inspect: - Definition, prerequisite, and contrast concept links - The equation or code witness that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

Open source object
concept/concept-notebook/attention-transformers/swiglu concept:attention-transformers/swiglu