Continuous Function

Reinforcement Learning

Bellman Equations

Bellman equations make value recursive: a state's value must equal immediate reward plus discounted value flowing back from the next states.

status: reviewimportance: criticaldifficulty 4/5math: graduateread: 22mlive demo
Back to Reinforcement LearningNext: Dynamic Programming and Value Iteration (review)

Concept Structure

Bellman Equations

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

1prerequisites
4next concepts
2related links

Learner Contract

What this page should let you do.

You are here becauseBellman equations make value recursive: a state's value must equal immediate reward plus discounted value flowing back from the next states.

This Reinforcement Learning concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Test the linkManipulate one control and predict the visible change.Then continue to Dynamic Programming and Value Iteration (review)

Claim/source review status

Substantive review recorded

1/1 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.
Claims1/1 reviewed
Sources4 cited
Codeattached
Demolive
Reviewed2026-07-03
Updatedpage 2026-07-03

Learning item flow

4/4 sections readyAsk about thisResearch room
ConceptBellman EquationsReinforcement LearningEquationBellman Equations equation 1Exact equationCodeBellman Equations runnable code 1Exact runnable codeDemoBellman Equations interactive demoInteractive demoClaimIn a finite discounted MDP, the Bellman expectation backup for a fixe...Exact claim checkSourceReinforcement Learning: An Introduction, Second EditionExact source
ConceptBellman EquationsReinforcement Learning
4 sources attachedLocal snapshot ready
concept:reinforcement-learning/bellman-equations
CodenearbyPredictbefore revealRoomitem handoff
01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.
01

01

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

The MDP formalism gave us the one-step contract:

current state, action, transition probabilities, and rewards.

Bellman equations ask the next question:

If the future already has value estimates, what should this state's value be?

The answer is recursive. A state is valuable when its next transition gives reward now and lands in states that are valuable later.

That is the Bellman idea:

  • reward flows forward from the transition;
  • value flows backward from the next state;
  • discount γ\gamma decides how much future value matters.

For a fixed policy π\pi, the Bellman expectation equation is a consistency equation. It does not search for the best action. It asks whether the value function agrees with the policy's own behavior.

For control, the Bellman optimality equation changes one operator: instead of averaging actions with π(a∣s)\pi(a\mid s), it takes the best action. That distinction matters. Policy evaluation answers "what is this policy worth?" Optimality answers "what would the best action be worth?"

This page keeps those two backups next to each other, but it does not run value iteration yet. First we need to feel one backup.

02

02

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt
Equation 1Vπ(s)=Eπ[∑k=0∞γkRt+k+1∣St=s].V^\pi(s) = \mathbb{E}_\pi\left[ \sum_{k=0}^{\infty}\gamma^k R_{t+k+1} \mid S_t=s \right].Equation 2Vπ(s)=∑a∈Aπ(a∣s)∑s′∈SP(s′∣s,a)[R(s,a,s′)+γVπ(s′)].V^\pi(s) = \sum_{a\in\mathcal{A}}\pi(a\mid s) \sum_{s'\in\mathcal{S}} P(s'\mid s,a) \left[ R(s,a,s')+\gamma V^\pi(s') \right].

Let M=(S,A,P,R,γ,ρ0)\mathcal{M}=(\mathcal{S},\mathcal{A},P,R,\gamma,\rho_0) be a finite discounted MDP, with discount γ∈[0,1)\gamma\in[0,1).

For a policy π(a∣s)\pi(a\mid s), the state-value function is

Vπ(s)=Eπ[∑k=0∞γkRt+k+1∣St=s].V^\pi(s) = \mathbb{E}_\pi\left[ \sum_{k=0}^{\infty}\gamma^k R_{t+k+1} \mid S_t=s \right].

The Bellman expectation equation says this infinite return can be split into one reward plus the discounted value of the next state:

Vπ(s)=∑a∈Aπ(a∣s)∑s′∈SP(s′∣s,a)[R(s,a,s′)+γVπ(s′)].V^\pi(s) = \sum_{a\in\mathcal{A}}\pi(a\mid s) \sum_{s'\in\mathcal{S}} P(s'\mid s,a) \left[ R(s,a,s')+\gamma V^\pi(s') \right].

Read the equation from inside out.

First, for a particular transition (s,a,s′)(s,a,s'), compute the bracket:

R(s,a,s′)+γVπ(s′).R(s,a,s')+\gamma V^\pi(s').

Then average those brackets over next states using P(s′∣s,a)P(s'\mid s,a). Then average the action rows using π(a∣s)\pi(a\mid s).

For an arbitrary value estimate VV, the one-step Bellman expectation backup is an operator:

(TπV)(s)=∑aπ(a∣s)∑s′P(s′∣s,a)[R(s,a,s′)+γV(s′)].(\mathcal{T}^{\pi}V)(s) = \sum_a\pi(a\mid s)\sum_{s'}P(s'\mid s,a) \left[ R(s,a,s')+\gamma V(s') \right].

The true value VπV^\pi is the fixed point:

Vπ=TπVπ.V^\pi = \mathcal{T}^{\pi}V^\pi.

The Bellman optimality backup removes the fixed policy and chooses the best action:

(T⋆V)(s)=max⁡a∈A∑s′∈SP(s′∣s,a)[R(s,a,s′)+γV(s′)].(\mathcal{T}^{\star}V)(s) = \max_{a\in\mathcal{A}} \sum_{s'\in\mathcal{S}}P(s'\mid s,a) \left[ R(s,a,s')+\gamma V(s') \right].

The optimal value function V⋆V^\star satisfies

V⋆=T⋆V⋆.V^\star = \mathcal{T}^{\star}V^\star.

This page is about the backup itself. Value iteration repeatedly applies T⋆\mathcal{T}^{\star}; temporal-difference methods learn related targets from sampled transitions.

03

03

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt
Runnable code 1states = ["Start", "Practice", "Confused", "Mastered"] V = {"Start": 0.40, "Practice": 1.20,...python
states = ["Start", "Practice", "Confused", "Mastered"]
V = {"Start": 0.40, "Practice": 1.20, "Confused": -0.60, "Mastered": 0.0}
gamma = 0.90
pi = {"drill": 0.70, "skim": 0.30}
P = {
    "drill": [("Practice", 0.62, 1.0), ("Confused", 0.23, -0.5), ("Mastered", 0.15, 3.0)],
    "skim": [("Start", 0.30, 0.1), ("Practice", 0.45, 0.5), ("Confused", 0.25, -0.2)],
}

def action_backup(action, gamma=gamma):
    total = 0.0
    for next_state, prob, reward in P[action]:
        bracket = reward + gamma * V[next_state]
        total += prob * bracket
    return total

def expectation_backup(policy, gamma=gamma):
    total = 0.0
    for action, weight in policy.items():
        total += weight * action_backup(action, gamma)
    return total

print("T^pi V(Start):", round(expectation_backup(pi), 3))
print("T^pi with gamma=0:", round(expectation_backup(pi, gamma=0.0), 3))
print("T^* V(Start):", round(max(action_backup(a) for a in pi), 3))

The same transition kernel from the MDP page is still here. The new ingredient is that every next state carries a value estimate, so each transition contributes

P(s′∣s,a)[R(s,a,s′)+γV(s′)].P(s'\mid s,a)\left[R(s,a,s')+\gamma V(s')\right].
04

04

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Use the Bellman Backup Lab as a value-flow prediction check.

Pick a case, then predict which backup is being computed:

  • immediate reward only;
  • reward plus discounted next value under a fixed policy;
  • or a max over action rows.

Before reveal, the backup value and contribution ledger stay locked. After reveal, inspect which terms were averaged, which future values flowed backward, and whether the update was policy evaluation or control.

Live Concept Demo

Explore Bellman Equations

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 4/5graduatecode-aligned
Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what Bellman Equations should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

Bellman equations make value recursive: a state's value must equal immediate reward plus discounted value flowing back from the next states.

Prediction open01 / Intuition
Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Bellman Equations should make visible.

Visual Inquiry

Make the image answer a mathematical question

Bellman equations make value recursive: a state's value must equal immediate reward plus discounted value flowing back from the next states.

4/4 stages readyLive demo connected
Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Bellman Equations easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

book · 2018Reinforcement Learning: An Introduction, Second EditionRichard S. Sutton and Andrew G. Barto

Canonical source for value functions, Bellman expectation equations, Bellman optimality equations, and bootstrapping from one-step dynamics.

Open source
course-notes · 2026Stanford CS234: Lecture 2, MDPs, Bellman Equations, and PlanningStanford CS234

Graduate RL course source for value functions, Bellman equations, policy evaluation, and planning with known dynamics.

Open source
course-notes · 2026UC Berkeley CS188: Value IterationUC Berkeley CS188

Course-note source for recursive value updates, discounted future value, and the max-over-actions optimality backup.

Open source
book · 2026Dive into Deep Learning: Value IterationDive into Deep Learning

Open textbook source for Bellman optimality and value-iteration framing in tabular reinforcement learning.

Open source

Claim Review

Bellman equations make value recursive: a state's value must equal immediate reward plus discounted value flowing back from the next states.

Status1 substantive review recorded

Claims without a substantive review badge still need exact source-support review.

Sources4 references

sutton-barto-2018-rl, stanford-cs234-bellman, berkeley-cs188-value-iteration, d2l-value-iteration

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedIn a finite discounted MDP, the Bellman expectation backup for a fixed policy computes immediate reward plus discounted expected next-state value, while the Bellman optimality backup replaces the policy average over actions with a max over actions.Claim metadata: source checked

The sources support discounted value functions, one-step Bellman recursion, the fixed-policy expectation backup, and the optimality backup that selects the best action under known dynamics.

Sources: Reinforcement Learning: An Introduction, Second Edition, Stanford CS234: Lecture 2, MDPs, Bellman Equations, and Planning, UC Berkeley CS188: Value Iteration, Dive into Deep Learning: Value IterationFinite tabular teaching MDP only; excludes convergence proofs, contraction mapping details, asynchronous dynamic programming, stochastic approximation, temporal-difference learning, continuous state/action spaces, function approximation, and off-policy data.A bounded review summary is present; still check caveats and exact reference scope.

Checked Sutton and Barto for value functions and Bellman expectation/optimality equations; Stanford CS234 for graduate policy-evaluation and planning framing; Berkeley CS188 for recursive value updates and optimality backups; and D2L for value-iteration/Bellman optimality context. GPT Pro publication critique remains pending because 127.0.0.1:51672 is unavailable.

Reviewer: codex-local-source-review; reviewed 2026-07-03
source-span-sutton-barto-2018-rlsource-span-stanford-cs234-bellmansource-span-berkeley-cs188-value-iterationsource-span-d2l-value-iterationEquation 1Equation 2Runnable code 1Interactive demo

Source support candidates

book 2018Reinforcement Learning: An Introduction, Second Edition

Canonical source for value functions, Bellman expectation equations, Bellman optimality equations, and bootstrapping from one-step dynamics.

course-notes 2026Stanford CS234: Lecture 2, MDPs, Bellman Equations, and Planning

Graduate RL course source for value functions, Bellman equations, policy evaluation, and planning with known dynamics.

course-notes 2026UC Berkeley CS188: Value Iteration

Course-note source for recursive value updates, discounted future value, and the max-over-actions optimality backup.

book 2026Dive into Deep Learning: Value Iteration

Open textbook source for Bellman optimality and value-iteration framing in tabular reinforcement learning.

Mechanism checks

Equation 1
Vπ(s)=Eπ[∑k=0∞γkRt+k+1∣St=s].V^\pi(s) = \mathbb{E}_\pi\left[ \sum_{k=0}^{\infty}\gamma^k R_{t+k+1} \mid S_t=s \right].
Equation 2
Vπ(s)=∑a∈Aπ(a∣s)∑s′∈SP(s′∣s,a)[R(s,a,s′)+γVπ(s′)].V^\pi(s) = \sum_{a\in\mathcal{A}}\pi(a\mid s) \sum_{s'\in\mathcal{S}} P(s'\mid s,a) \left[ R(s,a,s')+\gamma V^\pi(s') \right].
Runnable code 1states = ["Start", "Practice", "Confused", "Mastered"] V = {"Start": 0.40, "Practice": 1.20,...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

Bellman equations make value recursive: a state's value must equal immediate reward plus discounted value flowing back from the next states.

Readiness0/3 checks ready
Predict

Before touching the demo, predict one visible change that should happen in Bellman Equations.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Grounded research drawerClose
ConceptBellman EquationsReinforcement Learning
Runnable code comparisonBellman Equations runnable code 1states = ["Start", "Practice", "Confused", "Mastered"] V = {"Start": 0.40, "Practice": 1.20,...Prediction before revealBellman Equations interactive demoManipulate one control and predict the visible change.
Grounded room questionWhat is the smallest example that makes Bellman Equations click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.
Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptReinforcement Learning

Bellman Equations

Attached question

What is the smallest example that makes Bellman Equations click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action
Local action draft

This draft stays in this browser, attached to the selected learning item.

No local draft saved.
Evidence to inspect
  • References to inspect: attached references on this page.
  • Definition, prerequisite, and contrast concept links
  • The equation or runnable code that makes the concept operational
  • One demo state that shows the invariant instead of a slogan
What would resolve this
  • The learner can state the mechanism in their own words
  • The learner can name the prerequisite that would repair confusion
  • The learner can predict how the mechanism changes under one perturbation
Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - Bellman Equations Selected item key: recorded for copy. Context: Reinforcement Learning Page anchor: recorded for copy. Open question: What is the smallest example that makes Bellman Equations click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context
concept/concept-notebook/reinforcement-learning/bellman-equations concept:reinforcement-learning/bellman-equations