Reinforcement Learning

Function Approximation in RL

Function approximation replaces a Q-table with features and weights, so one TD update can generalize to nearby state-action pairs and can become unstable when bootstrapping, off-policy data, and approximation combine.

status: reviewimportance: importantdifficulty 4/5math: graduateread: 22mlive demo

Back to Reinforcement Learning Next: Deep Q-Learning (review)

Concept Structure

Function Approximation in RL

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

2prerequisites

2next concepts

3related links

Learner Contract

What this page should let you do.

You are here becauseFunction approximation replaces a Q-table with features and weights, so one TD update can generalize to nearby state-action pairs and can become unstable when bootstrapping, off-policy data, and approximation combine.

This Reinforcement Learning concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

Before thisQ-Learning and SARSA (review)

2 prerequisites listed; refresh them before leaning on the math or code.

Q-Learning and SARSA (review)MLPs and Universal Approximation (review)

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Then go nextDeep Q-Learning (review)

Follow this edge after making one prediction here; the next page should reuse the result, not restart the route.

Test the linkManipulate one control and predict the visible change.Then continue to Deep Q-Learning (review)

Claim/source review status

Substantive review recorded

1/1 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.

Claims1/1 reviewed

Sources2 cited

Codeattached

Demolive

Reviewed2026-07-03

Updatedpage 2026-07-03

Learning item flow

4/4 sections readyAsk about this Research room

ConceptFunction Approximation in RLReinforcement Learning EquationFunction Approximation in RL equation 1Exact equation CodeFunction Approximation in RL runnable code 1Exact runnable code DemoFunction Approximation in RL interactive demoInteractive demo ClaimWith a differentiable action-value approximator q_hat(s,a;w), semi-gr...Exact claim check SourceReinforcement Learning: An Introduction, Second EditionExact source

ConceptFunction Approximation in RLReinforcement Learning

2 sources attachedLocal snapshot ready

concept:reinforcement-learning/rl-function-approximation

Codenearby Predictbefore reveal Roomitem handoff

01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

Q-learning and SARSA used a table: one number for each state-action pair. That is clean, but it breaks the moment the state space becomes large, continuous, or visually rich.

Function approximation asks a different question:

Instead of storing every $Q(s,a)$ separately, what features of $(s,a)$ should share credit?

If the agent updates the value of Practice + Drill, a table changes only that cell. A feature-based approximator can also move Practice + Ask because both share the Practice-state feature, or Confused + Drill because both share the Drill-action feature. That is the power and the danger: one transition can generalize beyond itself.

The simplest version is linear. A feature map $\phi(s,a)$ turns a state-action pair into a vector, and weights $w$ turn that vector into a value estimate. The update looks like TD learning, but it changes weights, not table cells.

The caution is not optional. Bootstrapping already uses an estimate as a target. Off-policy learning can train on data from a different policy than the one being improved. Function approximation spreads each update through shared features. Sutton and Barto call the combination of these three ingredients the deadly triad because it can cause oscillation or divergence. Deep Q-learning is powerful partly because it adds engineering devices around this problem, not because the problem vanished.

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt

Equation 1 $\hat q(s,a;w)=w^\top \phi(s,a),$ Equation 2 $S_t,A_t,R_{t+1},S_{t+1},$

Let $\phi(s,a)\in\mathbb R^d$ be a feature vector for state $s$ and action $a$ . A linear action-value approximator is

\hat q(s,a;w)=w^\top \phi(s,a),

where $w\in\mathbb R^d$ is learned. More generally, $\hat q(s,a;w)$ can be a neural network, but the linear case makes the credit-sharing mechanism visible.

For a sampled transition

S_t,A_t,R_{t+1},S_{t+1},

a SARSA-style target with approximation is

U_t = R_{t+1}+\gamma \hat q(S_{t+1},A_{t+1};w_t),

while a Q-learning-style target is

U_t = R_{t+1} +\gamma \max_{a'} \hat q(S_{t+1},a';w_t).

Both can be written as a semi-gradient update:

w_{t+1} = w_t + \alpha \left[ U_t-\hat q(S_t,A_t;w_t) \right] \nabla_w \hat q(S_t,A_t;w_t).

For the linear model, the gradient is just the active feature vector:

\nabla_w \hat q(S_t,A_t;w_t)=\phi(S_t,A_t),

so the update becomes

w_{t+1} = w_t + \alpha \delta_t \phi(S_t,A_t), \qquad \delta_t=U_t-\hat q(S_t,A_t;w_t).

This equation explains generalization. Another pair $(s',a')$ changes by

\hat q(s',a';w_{t+1})-\hat q(s',a';w_t) = \alpha\delta_t\,\phi(s',a')^\top\phi(S_t,A_t).

If the two feature vectors overlap, the unvisited pair moves. If they are orthogonal one-hot table features, it does not. The table is a special case of function approximation where every pair has its own isolated feature.

The word semi-gradient matters. When $U_t$ contains $\hat q(\cdot;w_t)$ , the target itself depends on the current weights. The update treats the sampled target as fixed for the step rather than taking the full gradient through both sides of a Bellman error. That is useful and common, but it is exactly why convergence guarantees are narrower than the tabular story.

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt

Runnable code 1import numpy as np alpha, gamma = 0.25, 0.90 w = np.array([0.45, 0.20, 0.25]) # state feature...python

import numpy as np

alpha, gamma = 0.25, 0.90
w = np.array([0.45, 0.20, 0.25])  # state feature, action feature, pair detail

phi = {
    ("practice", "drill"): np.array([1.0, 1.0, 1.0]),
    ("practice", "ask"):   np.array([1.0, 0.0, 0.0]),
    ("confused", "drill"): np.array([0.0, 1.0, 0.0]),
    ("confused", "ask"):   np.array([0.0, 0.0, 1.0]),
}

transition = ("practice", "drill", 0.50, "confused")
next_actions = ["drill", "ask"]

s, a, reward, s_next = transition
q_current = w @ phi[(s, a)]
q_next = max(w @ phi[(s_next, a2)] for a2 in next_actions)
target = reward + gamma * q_next
td_error = target - q_current

w_new = w + alpha * td_error * phi[(s, a)]

for pair in phi:
    before = w @ phi[pair]
    after = w_new @ phi[pair]
    print(pair, "before", round(before, 3), "after", round(after, 3))

The visited pair receives the biggest change because all three of its features are active. Pairs that share only the state or action feature move too. A one-hot table would not do that.

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Use the RL Function Approximation Lab to predict what one TD update changes.

Choose a feature design, inspect the active feature vector and the TD target, then commit to whether the update should change only the visited pair, move feature-sharing neighbors too, or be treated as a caution case. After reveal, the lab shows the TD error, weight deltas, before/after estimates for nearby state-action pairs, and the reason the off-policy bootstrapped case needs extra care.

Live Concept Demo

Explore Function Approximation in RL

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 4/5graduatecode-aligned

Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what Function Approximation in RL should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

Prediction open01 / Intuition

Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Function Approximation in RL should make visible.

Visual Inquiry

Make the image answer a mathematical question

4/4 stages readyLive demo connected

Visual cueWhich visible object should carry the first intuition?

Inspection depth2/4

Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Function Approximation in RL easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

book · 2018Reinforcement Learning: An Introduction, Second EditionRichard S. Sutton and Andrew G. Barto

Canonical source for approximate value functions, feature vectors, semi-gradient prediction/control updates, action-value approximation, and the deadly-triad warning.

Open source

course-notes · 2026Stanford CS234: Lecture 4, Model-Free Control and Function ApproximationStanford CS234

Graduate RL source for TD and control with value-function approximation, SARSA/Q-learning approximation targets, and instability from bootstrapping, off-policy learning, and approximation.

Open source

Claim Review

Function approximation replaces a Q-table with features and weights, so one TD update can generalize to nearby state-action pairs and can become unstable when bootstrapping, off-policy data, and approximation combine.

Status1 substantive review recorded

Claims without a substantive review badge still need exact source-support review.

Sources2 references

sutton-barto-2018-rl, stanford-cs234-function-approximation-2026

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedWith a differentiable action-value approximator q_hat(s,a;w), semi-gradient TD control updates weights by a TD error times grad_w q_hat(s,a;w); shared features make a single visited transition change estimates for other state-action pairs, and bootstrapping plus off-policy data plus approximation can be unstable.Claim metadata: source checked

The sources support feature-based approximate value functions, semi-gradient updates that treat bootstrapped targets as targets for the current step, action-value control updates for SARSA and Q-learning, and the instability risk when function approximation, bootstrapping, and off-policy learning are combined.

Sources: Reinforcement Learning: An Introduction, Second Edition, Stanford CS234: Lecture 4, Model-Free Control and Function ApproximationLinear feature-based teaching lab only; excludes nonlinear neural-network optimization, replay buffers, target networks, Double DQN, distributional RL, eligibility traces, emphatic TD, gradient-TD methods, and formal convergence proofs.A bounded review summary is present; still check caveats and exact reference scope.

Checked Sutton/Barto Chapters 9-11 for approximate values, feature vectors, semi-gradient methods, action-value approximation, off-policy divergence, and the deadly triad. Checked Stanford CS234 Lecture 4 pages 52-60 for TD/control with q_hat(s,a;w), approximate SARSA/Q-learning targets, and instability warnings. GPT Pro critique remains pending because 127.0.0.1:51672 is unavailable.

Reviewer: codex-local-source-review; reviewed 2026-07-03

source-span-sutton-barto-2018-rl source-span-stanford-cs234-function-approximation-2026 Equation 1 Equation 2 Runnable code 1 Interactive demo

Source support candidates

book 2018Reinforcement Learning: An Introduction, Second Edition

Canonical source for approximate value functions, feature vectors, semi-gradient prediction/control updates, action-value approximation, and the deadly-triad warning.

course-notes 2026Stanford CS234: Lecture 4, Model-Free Control and Function Approximation

Graduate RL source for TD and control with value-function approximation, SARSA/Q-learning approximation targets, and instability from bootstrapping, off-policy learning, and approximation.

Mechanism checks

Equation 1

\hat q(s,a;w)=w^\top \phi(s,a),

Equation 2

S_t,A_t,R_{t+1},S_{t+1},

Runnable code 1import numpy as np alpha, gamma = 0.25, 0.90 w = np.array([0.45, 0.20, 0.25]) # state feature...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

Readiness0/3 checks ready

Predict

Before touching the demo, predict one visible change that should happen in Function Approximation in RL.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Your answer canvas

Local checks

Claim

A concrete answer is on the canvas.

Mechanism

The answer names why the claim should hold.

Bridge

It touches the page context or a neighboring idea.

Misconception check

Grounded research drawerClose

ConceptFunction Approximation in RLReinforcement Learning

Runnable code comparisonFunction Approximation in RL runnable code 1alpha, gamma = 0.25, 0.90 Prediction before revealFunction Approximation in RL interactive demoManipulate one control and predict the visible change.

Grounded room questionWhat is the smallest example that makes Function Approximation in RL click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptReinforcement Learning

Function Approximation in RL

Attached question

What is the smallest example that makes Function Approximation in RL click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action

Local action draft

This draft stays in this browser, attached to the selected learning item.

Draft noteNext action

No local draft saved.

Evidence to inspect

References to inspect: attached references on this page.
Definition, prerequisite, and contrast concept links
The equation or runnable code that makes the concept operational
One demo state that shows the invariant instead of a slogan

What would resolve this

The learner can state the mechanism in their own words
The learner can name the prerequisite that would repair confusion
The learner can predict how the mechanism changes under one perturbation

Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - Function Approximation in RL Selected item key: recorded for copy. Context: Reinforcement Learning Page anchor: recorded for copy. Open question: What is the smallest example that makes Function Approximation in RL click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context

concept/concept-notebook/reinforcement-learning/rl-function-approximation
concept:reinforcement-learning/rl-function-approximation

Learning Map

Before / Now / Try / Next

BeforeQ-Learning and SARSA (review)

NowIntuition -> Math -> Code -> Demo

TryManipulate one control and predict the visible change.

NextDeep Q-Learning (review)

Intuitionready
Mathready
Codeready
Interactive Demoready

AI Companion

Ask beside the selected object

Your question

GoalComfortStyleStuck on

Context prompt

You are my AI learning companion for Continuous Function. Current context: Reinforcement Learning concept. Learning surface: Function Approximation in RL. What this page says: Function approximation replaces a Q-table with features and weights, so one TD update can generalize to nearby state-action pairs and can become unstable when bootstrapping, off-policy data, and approximation combine. Current section: Intuition, math, code, and interactive demo. Suggested next step: Manipulate one control and predict the visible change.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Explain the central idea in plain language, then restate it with the exact math objects from the page. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.

Domain

Reinforcement Learning

reinforcement-learningfunction-approximationsemi-gradientvalue-function-approximationdeadly-triad

Prerequisites

Q-Learning and SARSA (review)MLPs and Universal Approximation (review)

Leads To

Deep Q-Learning (review)Actor Critic (planned)

Temporal-Difference Learning (review)Policy Gradient (review)MLPs and Universal Approximation (review)

Within this domain