Continuous Function

Machine Learning

MLPs and Universal Approximation

An MLP turns nonlinear hidden units into a weighted feature sum; universal approximation is capacity under assumptions, not a guarantee that training or generalization will succeed.

status: reviewimportance: criticaldifficulty 3/5math: graduateread: 18mlive demo
Back to Machine LearningNext: Backpropagation

Concept Structure

MLPs and Universal Approximation

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

4prerequisites
5next concepts
3related links

Learner Contract

What this page should let you do.

You are here becauseAn MLP turns nonlinear hidden units into a weighted feature sum; universal approximation is capacity under assumptions, not a guarantee that training or generalization will succeed.

This Machine Learning concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Then go nextBackpropagation

Follow this edge after making one prediction here; the next page should reuse the result, not restart the route.

Test the linkManipulate one control and predict the visible change.Then continue to Backpropagation

Claim/source review status

Substantive review recorded

1/1 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.
Claims1/1 reviewed
Sources5 cited
Codeattached
Demolive
Reviewed2026-07-01
Updatedpage 2026-07-01

Learning item flow

4/4 sections readyAsk about thisResearch room
ConceptMLPs and Universal ApproximationMachine LearningEquationMLPs and Universal Approximation equation 1Exact equationCodeMLPs and Universal Approximation runnable code 1Exact runnable codeDemoMLPs and Universal Approximation interactive demoInteractive demoClaimA one-hidden-layer MLP builds a weighted sum of nonlinear hidden feat...Exact claim checkSourceDeep Learning, Chapter 6: Deep Feedforward NetworksExact source
ConceptMLPs and Universal ApproximationMachine Learning
5 sources attachedLocal snapshot ready
concept:machine-learning/mlps-universal-approximation
CodenearbyPredictbefore revealRoomitem handoff
01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.
01

01

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

You are here because "neural network" should not feel like a magical box. The simplest serious network, the multilayer perceptron, does one powerful thing: it learns hidden features, then adds them together.

Before this, know functions, least-squares fitting, and derivatives. By the end, you should be able to explain why nonlinear hidden units make MLPs more expressive than linear regression, what universal approximation actually promises, and why that promise is not the same as successful training.

A linear model gives every input one flat set of slopes. If the target bends one way on the left and another way on the right, a single line cannot follow both. An MLP first creates hidden features such as "turn on after this threshold" or "saturate around this region." The output layer then forms a weighted sum of those features.

That weighted sum is the useful mental model. A hidden unit is not a neuron mysticism object. It is a learned nonlinear feature. Width gives the model more features to combine. Depth lets features compose into features of features.

Universal approximation says something precise and easy to overstate. In Cybenko's classic form, continuous sigmoidal hidden units can uniformly approximate continuous functions on a compact cube. In the Leshno-Lin-Pinkus-Schocken form, locally bounded piecewise-continuous nonpolynomial activations with thresholds/biases give a broad approximation condition. Neither result says gradient descent will find those weights, that a finite dataset identifies the right function, that the needed width is small, or that the approximation generalizes.

The practical lesson is therefore two-sided:

  • MLPs have enough representational capacity to bend beyond linear models.
  • Capacity alone is not a learning guarantee. Optimization, data, regularization, initialization, and architecture still decide what happens.
02

02

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt
Equation 1h=ϕ(W1x+b1),h=\phi(W_1x+b_1),Equation 2f^(x)=W2h+b2,\hat f(x)=W_2h+b_2,

For a one-hidden-layer scalar-output MLP, let x∈Rdx\in\mathbb R^d. Define

h=ϕ(W1x+b1),h=\phi(W_1x+b_1),

where W1∈Rm×dW_1\in\mathbb R^{m\times d}, b1∈Rmb_1\in\mathbb R^m, ϕ\phi is applied elementwise, and mm is the hidden width. The prediction is

f^(x)=W2h+b2,\hat f(x)=W_2h+b_2,

where W2∈R1×mW_2\in\mathbb R^{1\times m} and b2∈Rb_2\in\mathbb R. In one dimension this can be written as

f^(x)=c+∑j=1maj ϕ(wjx+βj).\hat f(x)=c+\sum_{j=1}^m a_j\,\phi(w_jx+\beta_j).

This is the "weighted sum of hidden features" picture. Each ϕ(wjx+βj)\phi(w_jx+\beta_j) is a feature curve. The output weights aja_j decide how strongly that feature contributes.

The nonlinearity is essential. If ϕ(t)=t\phi(t)=t, then two affine layers collapse:

f^(x)=W2(W1x+b1)+b2=(W2W1)x+(W2b1+b2).\hat f(x)=W_2(W_1x+b_1)+b_2=(W_2W_1)x+(W_2b_1+b_2).

That is just another affine map. Stacking linear layers without nonlinear activations does not buy curved decision boundaries or curved regression functions.

One common universal-approximation template says: for a compact set K⊂RdK\subset\mathbb R^d, an activation satisfying the theorem's conditions, and any continuous function f:K→Rf:K\to\mathbb R, for every ε>0\varepsilon>0 there exists a width mm and parameters (aj,wj,βj,c)(a_j,w_j,\beta_j,c) such that

sup⁡x∈K∣f(x)−(c+∑j=1majϕ(wjTx+βj))∣<ε.\sup_{x\in K}\left|f(x)-\left(c+\sum_{j=1}^m a_j\phi(w_j^{\mathsf T}x+\beta_j)\right)\right|<\varepsilon.

Read the quantifiers carefully. The theorem says there exist parameters. It does not specify how many units are practical, how to find the parameters from finite noisy data, or whether the learned function behaves well away from the training distribution. Biases or thresholds are part of the usual theorem setup; remove them and the approximation claim can change.

03

03

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt
Runnable code 1import numpy as np # Fixed ReLU-hinge features fitted by least squares. # This is a function-...python
import numpy as np

# Fixed ReLU-hinge features fitted by least squares.
# This is a function-approximation witness, not SGD training.
def f_star(x):
    return 0.75 * np.sin(2 * np.pi * x) + 0.25 * np.cos(5 * np.pi * x)

def relu(z):
    return np.maximum(z, 0.0)

def design(x, knots):
    cols = [np.ones_like(x), x]
    cols += [relu(x - k) for k in knots]
    return np.column_stack(cols)

x_train = np.linspace(-1, 1, 45)
x_probe = np.linspace(-1, 1, 201)
y_train = f_star(x_train)
y_probe = f_star(x_probe)

for name, knots in {
    "linear": [],
    "two_hinges": [-0.35, 0.35],
    "eight_hinges": np.linspace(-0.85, 0.85, 8),
}.items():
    X = design(x_train, knots)
    beta = np.linalg.solve(X.T @ X + 1e-4 * np.eye(X.shape[1]), X.T @ y_train)
    pred = design(x_probe, knots) @ beta
    mse = np.mean((pred - y_probe) ** 2)
    print(name, "features:", X.shape[1], "probe_mse:", round(mse, 4))

The code mirrors the math. Each ReLU hinge is a hidden feature. The least-squares solve chooses only the output combination for a fixed feature bank. Real MLP training also learns the hidden weights and uses gradient-based optimization, so this witness is deliberately narrower than full neural-network training.

04

04

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Choose a model family and predict what will happen before revealing the approximation:

  • Still large error: the family cannot bend enough for this target.
  • Useful approximation: hidden features combine into a close curve on the domain.
  • Memorizes noise: the fitted curve chases sparse noisy samples and looks worse between them.

Before reveal, the demo hides the approximation line, hidden-feature contributions, exact train error, and exact probe error. After reveal, compare the target, weighted sum, error band, and caveat strip. The point is not that eight units is universal. The point is that nonlinear feature sums can create curvature, while universal approximation remains an existence theorem rather than a training or generalization guarantee.

Live Concept Demo

Explore MLPs and Universal Approximation

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 3/5graduatecode-aligned
Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what MLPs and Universal Approximation should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

An MLP turns nonlinear hidden units into a weighted feature sum; universal approximation is capacity under assumptions, not a guarantee that training or generalization will succeed.

Prediction open01 / Intuition
Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change MLPs and Universal Approximation should make visible.

Visual Inquiry

Make the image answer a mathematical question

An MLP turns nonlinear hidden units into a weighted feature sum; universal approximation is capacity under assumptions, not a guarantee that training or generalization will succeed.

4/4 stages readyLive demo connected
Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make MLPs and Universal Approximation easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

book · 2016Deep Learning, Chapter 6: Deep Feedforward NetworksGoodfellow, Bengio, and Courville

Frames MLPs as feedforward function approximators, compositions of functions, hidden units, and backprop-trained models.

Open source
book · 2026Dive into Deep Learning: Multilayer PerceptronsZhang, Lipton, Li, and Smola

Supports affine layers, activation functions, hidden representations, and the nonlinearity needed to avoid collapsing into a linear map.

Open source
course-notes · 2026CS231n: Neural Networks Part 1Stanford CS231n

Course-note bridge for neural network architecture, representational power, and practical caveats around universal approximation.

Open source
paper · 1989Approximation by Superpositions of a Sigmoidal FunctionCybenko

Classic theorem source for dense approximation of continuous functions on compact subsets using sigmoidal hidden units.

Open source
paper · 1993Multilayer Feedforward Networks with a Nonpolynomial Activation Function Can Approximate Any FunctionLeshno, Lin, Pinkus, and Schocken

Theorem-scope source tying universal approximation for standard feedforward networks to nonpolynomial activation behavior.

Open source

Claim Review

An MLP turns nonlinear hidden units into a weighted feature sum; universal approximation is capacity under assumptions, not a guarantee that training or generalization will succeed.

Status1 substantive review recorded

Claims without a substantive review badge still need exact source-support review.

Sources5 references

goodfellow-2016-mlp, d2l-2026-mlp, cs231n-neural-networks-1, cybenko-1989-approximation, leshno-1993-nonpolynomial

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedA one-hidden-layer MLP builds a weighted sum of nonlinear hidden features; under compact-domain and activation assumptions, sufficiently many hidden units can approximate continuous target functions, but this is an existence statement rather than a guarantee of trainability, sample efficiency, or generalization.Claim metadata: source checked

Deep Learning Book, D2L, and CS231n support nonlinear hidden features and collapse without activation; Cybenko supports one-hidden-layer compact-domain approximation for continuous sigmoids; Leshno et al. support nonpolynomial activation theorem scope and representation-not-learning caveat.

Sources: Deep Learning, Chapter 6: Deep Feedforward Networks, Dive into Deep Learning: Multilayer Perceptrons, CS231n: Neural Networks Part 1, Approximation by Superpositions of a Sigmoidal Function, Multilayer Feedforward Networks with a Nonpolynomial Activation Function Can Approximate Any FunctionFixed 1D ReLU-hinge least-squares toy; not SGD, sample-efficiency, finite-width rates, infinite-width theory, calibration, practical architecture choice, or generalization guarantee.A bounded review summary is present; still check caveats and exact reference scope.

Checked Deep Learning Book, D2L, CS231n, Cybenko, and Leshno/Lin/Pinkus/Schocken. The sources support MLPs as nonlinear hidden-feature function approximators and support compact-domain universal-approximation theorem scope; Leshno explicitly frames the theorem work as representation rather than learning. GPT Pro/Oracle publication critique is still pending.

Reviewer: codex-local-primary-source-audit; reviewed 2026-07-01
source-span-goodfellow-2016-mlpsource-span-d2l-2026-mlpsource-span-cs231n-neural-networks-1source-span-cybenko-1989-approximationsource-span-leshno-1993-nonpolynomialEquation 1Runnable code 1Interactive demo

Source support candidates

book 2016Deep Learning, Chapter 6: Deep Feedforward Networks

Frames MLPs as feedforward function approximators, compositions of functions, hidden units, and backprop-trained models.

book 2026Dive into Deep Learning: Multilayer Perceptrons

Supports affine layers, activation functions, hidden representations, and the nonlinearity needed to avoid collapsing into a linear map.

course-notes 2026CS231n: Neural Networks Part 1

Course-note bridge for neural network architecture, representational power, and practical caveats around universal approximation.

paper 1989Approximation by Superpositions of a Sigmoidal Function

Classic theorem source for dense approximation of continuous functions on compact subsets using sigmoidal hidden units.

Mechanism checks

Equation 1
h=ϕ(W1x+b1),h=\phi(W_1x+b_1),
Equation 2
f^(x)=W2h+b2,\hat f(x)=W_2h+b_2,
Runnable code 1import numpy as np # Fixed ReLU-hinge features fitted by least squares. # This is a function-...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

An MLP turns nonlinear hidden units into a weighted feature sum; universal approximation is capacity under assumptions, not a guarantee that training or generalization will succeed.

Readiness0/3 checks ready
Predict

Before touching the demo, predict one visible change that should happen in MLPs and Universal Approximation.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Grounded research drawerClose
ConceptMLPs and Universal ApproximationMachine Learning
Runnable code comparisonMLPs and Universal Approximation runnable code 1cols = [np.ones_like(x), x]Prediction before revealMLPs and Universal Approximation interactive demoManipulate one control and predict the visible change.
Grounded room questionWhat is the smallest example that makes MLPs and Universal Approximation click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.
Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptMachine Learning

MLPs and Universal Approximation

Attached question

What is the smallest example that makes MLPs and Universal Approximation click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action
Local action draft

This draft stays in this browser, attached to the selected learning item.

No local draft saved.
Evidence to inspect
  • References to inspect: attached references on this page.
  • Definition, prerequisite, and contrast concept links
  • The equation or runnable code that makes the concept operational
  • One demo state that shows the invariant instead of a slogan
What would resolve this
  • The learner can state the mechanism in their own words
  • The learner can name the prerequisite that would repair confusion
  • The learner can predict how the mechanism changes under one perturbation
Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - MLPs and Universal Approximation Selected item key: recorded for copy. Context: Machine Learning Page anchor: recorded for copy. Open question: What is the smallest example that makes MLPs and Universal Approximation click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context
concept/concept-notebook/machine-learning/mlps-universal-approximation concept:machine-learning/mlps-universal-approximation