Machine Learning

Activation Functions and Gradient Flow

Activation functions choose both the nonlinearity and the local derivative gate that controls how gradient signal flows through a network.

status: reviewimportance: criticaldifficulty 3/5math: graduateread: 17mlive demo

Back to Machine Learning Next: MLPs and Universal Approximation (review)

Concept Structure

Activation Functions and Gradient Flow

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

2prerequisites

5next concepts

3related links

Learner Contract

What this page should let you do.

You are here becauseActivation functions choose both the nonlinearity and the local derivative gate that controls how gradient signal flows through a network.

This Machine Learning concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

Before thisFunctions (review)

2 prerequisites listed; refresh them before leaning on the math or code.

Functions (review)Derivatives

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Then go nextMLPs and Universal Approximation (review)

Follow this edge after making one prediction here; the next page should reuse the result, not restart the route.

Test the linkManipulate one control and predict the visible change.Then continue to MLPs and Universal Approximation (review)

Claim/source review status

Substantive review recorded

1/1 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.

Claims1/1 reviewed

Sources4 cited

Codeattached

Demolive

Reviewed2026-07-02

Updatedpage 2026-07-02

Learning item flow

4/4 sections readyAsk about this Research room

ConceptActivation Functions and Gradient FlowMachine Learning EquationActivation Functions and Gradient Flow equation 1Exact equation CodeActivation Functions and Gradient Flow runnable code 1Exact runnable code DemoActivation Functions and Gradient Flow interactive demoInteractive demo ClaimActivations shape nonlinear features and backprop gates: sigmoid/tanh...Exact claim check SourceDive into Deep Learning: Multilayer Perceptrons, Activation FunctionsExact source

ConceptActivation Functions and Gradient FlowMachine Learning

4 sources attachedLocal snapshot ready

concept:machine-learning/activation-functions

Codenearby Predictbefore reveal Roomitem handoff

01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

You are here because "use ReLU" is not an explanation. An activation function is a small scalar function placed between affine layers, but it has two jobs at once: it bends the forward computation and it gates the backward gradient.

Before this, know functions and derivatives. By the end, you should be able to compare sigmoid, tanh, ReLU, GELU, and SiLU/Swish by asking two questions:

What output shape does this activation create?
What local derivative will backprop multiply into the gradient?

Without a nonlinearity, stacked affine layers collapse into one affine layer. With nonlinear activations, hidden units become features and deep networks can build piecewise, saturating, or smoothly gated transformations. But the derivative is the part learners often miss. If a layer's local derivative is tiny, then upstream gradient is multiplied by a tiny number. Repeat that across layers and the signal can vanish before useful learning reaches early weights.

Sigmoid and tanh are smooth squashing functions. They are nearly linear around zero, but their tails flatten, so their derivatives go toward zero when pre-activations are large in magnitude. ReLU is different: it is exactly zero on the negative side and identity on the positive side. That makes positive-side gradient flow simple, but negative units can be silent. GELU and SiLU/Swish are modern smooth relatives: instead of a hard threshold, they weight the input by a smooth gate.

The practical lesson is not a ranking table. Activation choice interacts with initialization, normalization, optimizer, architecture, and task. The local lesson is more durable: every activation leaves a fingerprint in the derivative chain.

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt

Equation 1 $z = Wx + b,$ Equation 2 $h_i = \phi(z_i).$

Let a hidden layer receive a pre-activation vector

z = Wx + b,

where $x\in\mathbb R^d$ , $W\in\mathbb R^{m\times d}$ , $b\in\mathbb R^m$ , and $z\in\mathbb R^m$ . The activation is applied elementwise:

h_i = \phi(z_i).

If $L$ is the loss, the chain rule gives

\frac{\partial L}{\partial z_i} = \frac{\partial L}{\partial h_i}\,\phi'(z_i).

So $\phi'(z_i)$ is a local gradient gate. A forward plot without the derivative plot is only half the story.

Common choices:

\sigma(z)=\frac{1}{1+e^{-z}}, \qquad \sigma'(z)=\sigma(z)(1-\sigma(z)).

The maximum sigmoid derivative is $0.25$ at $z=0$ , and it tends to $0$ as $|z|$ grows.

\tanh'(z)=1-\tanh^2(z).

The tanh derivative reaches $1$ at $z=0$ , but it also tends to $0$ in both tails.

\operatorname{ReLU}(z)=\max(0,z), \qquad \operatorname{ReLU}'(z)= \begin{cases} 0,& z\le 0,\\ 1,& z>0. \end{cases}

This is why ReLU can pass positive-side gradients cleanly while also creating silent negative-side units.

For smooth modern activations, let $\Phi(z)$ be the standard Gaussian cumulative distribution function. GELU is

\operatorname{GELU}(z)=z\Phi(z),

and SiLU, often called Swish when written with a parameter, is

\operatorname{SiLU}(z)=z\sigma(z).

These functions multiply the input by a smooth gate instead of making a hard negative/positive cutoff.

For a depth- $k$ scalar chain with local pre-activations $z_1,\dots,z_k$ , the magnitude of the backward signal contains the product

\prod_{\ell=1}^{k}|\phi'(z_\ell)|.

That product is the whole reason derivative plots matter. Three sigmoid units in the saturated tail can shrink a gradient by orders of magnitude. Three positive ReLU units pass a product of $1$ . Three smooth gates sit somewhere between, depending on the pre-activations.

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt

Runnable code 1import math def sigmoid(z): return 1.0 / (1.0 + math.exp(-z)) def normal_cdf(z): return 0.5 *...python

import math

def sigmoid(z):
    return 1.0 / (1.0 + math.exp(-z))

def normal_cdf(z):
    return 0.5 * (1.0 + math.erf(z / math.sqrt(2.0)))

def normal_pdf(z):
    return math.exp(-0.5 * z * z) / math.sqrt(2.0 * math.pi)

activations = {
    "sigmoid": (lambda z: sigmoid(z), lambda z: sigmoid(z) * (1 - sigmoid(z))),
    "tanh": (math.tanh, lambda z: 1 - math.tanh(z) ** 2),
    "relu": (lambda z: max(0.0, z), lambda z: 1.0 if z > 0 else 0.0),
    "gelu": (lambda z: z * normal_cdf(z), lambda z: normal_cdf(z) + z * normal_pdf(z)),
    "silu": (lambda z: z * sigmoid(z), lambda z: sigmoid(z) + z * sigmoid(z) * (1 - sigmoid(z))),
}

chain = [3.0, 2.4, 1.8]
for name, (_, deriv) in activations.items():
    gates = [deriv(z) for z in chain]
    product = math.prod(gates)
    print(name, [round(g, 4) for g in gates], "chain:", f"{product:.3e}")

The code mirrors the math: backprop through repeated activations multiplies local derivatives. The numbers are a tiny witness, not a claim that one activation wins across architectures.

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Use the Activation Derivative Flow Lab to choose one activation and predict the most visible derivative behavior before revealing:

Saturates gradients: the derivative is small in the tails, so the chain product collapses.
Passes positive gradients: positive-side ReLU derivatives pass signal, while negative-side units are silent.
Smooth self-gate: GELU or SiLU weights the input by a smooth gate rather than a hard threshold.

Before reveal, the demo shows the activation family and the pre-activation probe but hides the exact local derivative, the three-layer chain product, and the ReLU dead-side fraction. After reveal, compare the output curve, derivative curve, and chain strip. The right question is not "which activation is best?" It is "what derivative gate did this layer just put into the learning signal?"

Live Concept Demo

Explore Activation Functions and Gradient Flow

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 3/5graduatecode-aligned

Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what Activation Functions and Gradient Flow should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

Activation functions choose both the nonlinearity and the local derivative gate that controls how gradient signal flows through a network.

Prediction open01 / Intuition

Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Activation Functions and Gradient Flow should make visible.

Visual Inquiry

Make the image answer a mathematical question

Activation functions choose both the nonlinearity and the local derivative gate that controls how gradient signal flows through a network.

4/4 stages readyLive demo connected

Visual cueWhich visible object should carry the first intuition?

Inspection depth2/4

Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Activation Functions and Gradient Flow easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

book · 2026Dive into Deep Learning: Multilayer Perceptrons, Activation FunctionsZhang, Lipton, Li, and Smola

Gives ReLU, sigmoid, tanh, GELU, and Swish formulas plus derivative behavior for the classical activations.

Open source

course-notes · 2026CS231n: Neural Networks Part 1, Commonly Used Activation FunctionsStanford CS231n

Pedagogical source for nonlinearity, sigmoid saturation, ReLU non-saturation, and dying-ReLU caveats.

Open source

paper · 2016Gaussian Error Linear Units (GELUs)Hendrycks and Gimpel

Primary source for GELU as x times the standard Gaussian CDF.

Open source

paper · 2017Searching for Activation FunctionsRamachandran, Zoph, and Le

Primary source for Swish as x times sigmoid(beta x), with empirical comparison caveats.

Open source

Claim Review

Activation functions choose both the nonlinearity and the local derivative gate that controls how gradient signal flows through a network.

Status1 substantive review recorded

Claims without a substantive review badge still need exact source-support review.

Sources4 references

d2l-2026-activation-functions, cs231n-neural-networks-1-activations, hendrycks-2016-gelu, ramachandran-2017-swish

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedActivations shape nonlinear features and backprop gates: sigmoid/tanh saturate with near-zero tail derivatives, ReLU passes positive-side derivatives and blocks negative-side ones, and GELU/Swish use smooth input gates.Claim metadata: source checked

D2L supports ReLU/sigmoid/tanh formulas and derivative behavior plus GELU/Swish definitions; CS231n supports nonlinearity necessity, saturation killing gradients, and dying-ReLU caveats; the GELU and Swish papers support the modern smooth activation formulas.

Sources: Dive into Deep Learning: Multilayer Perceptrons, Activation Functions, CS231n: Neural Networks Part 1, Commonly Used Activation Functions, Gaussian Error Linear Units (GELUs), Searching for Activation FunctionsToy scalar local-derivative and chain-product witness only; not a benchmark claim, optimizer recommendation, architecture ranking, finite-training guarantee, normalization analysis, or output-layer prescription.A bounded review summary is present; still check caveats and exact reference scope.

Checked D2L for activation formulas and derivatives, CS231n for saturation/dead-ReLU training caveats and the nonlinearity-collapse warning, Hendrycks/Gimpel for GELU = x Phi(x), and Ramachandran/Zoph/Le for Swish = x sigmoid(beta x). Demo is a local derivative-chain witness, not an architecture benchmark or optimizer claim. GPT Pro/Oracle publication critique remains pending.

Reviewer: codex-local-primary-source-audit; reviewed 2026-07-02

source-span-d2l-2026-activation-functions source-span-cs231n-neural-networks-1-activations source-span-hendrycks-2016-gelu source-span-ramachandran-2017-swish Equation 1 Equation 2 Runnable code 1 Interactive demo

Source support candidates

book 2026Dive into Deep Learning: Multilayer Perceptrons, Activation Functions

Gives ReLU, sigmoid, tanh, GELU, and Swish formulas plus derivative behavior for the classical activations.

course-notes 2026CS231n: Neural Networks Part 1, Commonly Used Activation Functions

Pedagogical source for nonlinearity, sigmoid saturation, ReLU non-saturation, and dying-ReLU caveats.

paper 2016Gaussian Error Linear Units (GELUs)

Primary source for GELU as x times the standard Gaussian CDF.

paper 2017Searching for Activation Functions

Primary source for Swish as x times sigmoid(beta x), with empirical comparison caveats.

Mechanism checks

Equation 1

z = Wx + b,

Equation 2

h_i = \phi(z_i).

Runnable code 1import math def sigmoid(z): return 1.0 / (1.0 + math.exp(-z)) def normal_cdf(z): return 0.5 *...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

Activation functions choose both the nonlinearity and the local derivative gate that controls how gradient signal flows through a network.

Readiness0/3 checks ready

Predict

Before touching the demo, predict one visible change that should happen in Activation Functions and Gradient Flow.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Your answer canvas

Local checks

Claim

A concrete answer is on the canvas.

Mechanism

The answer names why the claim should hold.

Bridge

It touches the page context or a neighboring idea.

Misconception check

Grounded research drawerClose

ConceptActivation Functions and Gradient FlowMachine Learning

Runnable code comparisonActivation Functions and Gradient Flow runnable code 1activations = {Prediction before revealActivation Functions and Gradient Flow interactive demoManipulate one control and predict the visible change.

Grounded room questionWhat is the smallest example that makes Activation Functions and Gradient Flow click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptMachine Learning

Activation Functions and Gradient Flow

Attached question

What is the smallest example that makes Activation Functions and Gradient Flow click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action

Local action draft

This draft stays in this browser, attached to the selected learning item.

Draft noteNext action

No local draft saved.

Evidence to inspect

References to inspect: attached references on this page.
Definition, prerequisite, and contrast concept links
The equation or runnable code that makes the concept operational
One demo state that shows the invariant instead of a slogan

What would resolve this

The learner can state the mechanism in their own words
The learner can name the prerequisite that would repair confusion
The learner can predict how the mechanism changes under one perturbation

Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - Activation Functions and Gradient Flow Selected item key: recorded for copy. Context: Machine Learning Page anchor: recorded for copy. Open question: What is the smallest example that makes Activation Functions and Gradient Flow click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context

concept/concept-notebook/machine-learning/activation-functions
concept:machine-learning/activation-functions

Learning Map

Before / Now / Try / Next

BeforeFunctions (review)

NowIntuition -> Math -> Code -> Demo

TryManipulate one control and predict the visible change.

NextMLPs and Universal Approximation (review)

Intuitionready
Mathready
Codeready
Interactive Demoready

AI Companion

Ask beside the selected object

Activation functions choose both the nonlinearity and the local derivative gate that controls how gradient signal flows through a network.

Your question

GoalComfortStyleStuck on

Context prompt

You are my AI learning companion for Continuous Function. Current context: Machine Learning concept. Learning surface: Activation Functions and Gradient Flow. What this page says: Activation functions choose both the nonlinearity and the local derivative gate that controls how gradient signal flows through a network. Current section: Intuition, math, code, and interactive demo. Suggested next step: Manipulate one control and predict the visible change.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Explain the central idea in plain language, then restate it with the exact math objects from the page. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.

Domain

Machine Learning

machine-learningneural-networksactivation-functionsgradientsmlp

Prerequisites

Functions (review)Derivatives

Leads To

MLPs and Universal Approximation (review)Weight Initialization: Xavier, He, and Residual Scaling (review)SwiGLU: Gated MLP Blocks in Transformers (review)RNNs and Vanishing/Exploding Gradients (review)LSTM and GRU Gates (review)

Backpropagation Gradient Descent Layer Normalization & RMSNorm

Within this domain