Reinforcement Learning

Deep Q-Learning

Deep Q-learning turns Q-learning into neural value learning by training on replay-buffer samples, bootstrapping from a frozen target network, and watching for max-target overestimation.

status: reviewimportance: importantdifficulty 4/5math: graduateread: 24mlive demo

Back to Reinforcement Learning

Concept Structure

Deep Q-Learning

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

2prerequisites

1next concepts

3related links

Learner Contract

What this page should let you do.

You are here becauseDeep Q-learning turns Q-learning into neural value learning by training on replay-buffer samples, bootstrapping from a frozen target network, and watching for max-target overestimation.

This Reinforcement Learning concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

Before thisQ-Learning and SARSA (review)

2 prerequisites listed; refresh them before leaning on the math or code.

Q-Learning and SARSA (review)Function Approximation in RL (review)

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Then go nextDouble Dqn (planned)

Follow this edge after making one prediction here; the next page should reuse the result, not restart the route.

Test the linkManipulate one control and predict the visible change.

Claim/source review status

Substantive review recorded

1/1 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.

Claims1/1 reviewed

Sources4 cited

Codeattached

Demolive

Reviewed2026-07-03

Updatedpage 2026-07-03

Learning item flow

4/4 sections readyAsk about this Research room

ConceptDeep Q-LearningReinforcement Learning EquationDeep Q-Learning equation 1Exact equation CodeDeep Q-Learning runnable code 1Exact runnable code DemoDeep Q-Learning interactive demoInteractive demo ClaimDeep Q-learning trains a neural action-value approximator with Q-lear...Exact claim check SourcePlaying Atari with Deep Reinforcement LearningExact source

ConceptDeep Q-LearningReinforcement Learning

4 sources attachedLocal snapshot ready

concept:reinforcement-learning/deep-q-learning

Codenearby Predictbefore reveal Roomitem handoff

01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

Tabular Q-learning had one number for every state-action pair. Function approximation let nearby pairs share credit. Deep Q-learning takes the next step: replace the table with a neural network that receives a state and outputs one estimated value for each discrete action.

That change is necessary for images, long observations, and large state spaces, but it also makes the old Q-learning update more fragile. The target contains a value estimate from the same kind of model that is being trained. Consecutive experience is highly correlated. The max over next actions can prefer whichever action is accidentally overestimated.

DQN is the classic stabilization recipe:

Put transitions into a replay buffer instead of learning only from the newest step.
Sample minibatches from that buffer so updates mix many past behaviors.
Use an online network for the value being updated.
Use a separate target network for the bootstrap value, copying online weights into it only every so often.

So the learner's mental picture should not be "a bigger Q-table." It should be a small training circuit:

sampled transition -> target network target -> online network loss -> occasional target copy

The target network does not make the target true. It makes the target less rapidly moving. Replay does not make data independent. It reduces the worst correlation by training from older and newer transitions together. These are engineering stabilizers wrapped around the same Bellman idea you already know.

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt

Equation 1 $Q(s,\cdot;\theta) = \left[ Q(s,a_1;\theta),\ldots,Q(s,a_m;\theta) \right],$ Equation 2 $(s_i,a_i,r_i,s'_i,d_i),$

Let $Q(s,a;\theta)$ be the online neural action-value function with parameters $\theta$ . In many DQN implementations the network receives $s$ and outputs a vector

Q(s,\cdot;\theta) = \left[ Q(s,a_1;\theta),\ldots,Q(s,a_m;\theta) \right],

one value per discrete action. A replay buffer stores transitions

(s_i,a_i,r_i,s'_i,d_i),

where $d_i=1$ means the sampled transition ended an episode. DQN samples a minibatch from this buffer and builds a target for each transition.

The target network has parameters $\theta^-$ , held fixed for several gradient steps. The one-step DQN target is

y_i = \begin{cases} r_i, & d_i=1, \\ r_i + \gamma \max_{a'} Q(s'_i,a';\theta^-), & d_i=0. \end{cases}

The online network is trained to make the value of the sampled action match this target:

L_i(\theta) = \left( y_i - Q(s_i,a_i;\theta) \right)^2.

A gradient step changes $\theta$ to reduce this squared TD error, while $\theta^-$ stays fixed during that step. Periodically, DQN copies

\theta^- \leftarrow \theta.

This is still off-policy Q-learning. The replay buffer can contain transitions produced by older behavior policies, while the target uses a greedy max over the target network. That max is useful, but it also creates a bias risk: when action values are noisy, $\max_a Q(s',a)$ tends to select positive errors. Double DQN addresses that by using the online network to choose the next action and the target network to evaluate it, but plain DQN is the concept to understand first.

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt

Runnable code 1import torch gamma = 0.90 # A minibatch of three replay-buffer transitions. actions = torch.t...python

import torch

gamma = 0.90

# A minibatch of three replay-buffer transitions.
actions = torch.tensor([1, 0, 2])
rewards = torch.tensor([0.40, -0.10, 1.20])
dones = torch.tensor([0.0, 0.0, 1.0])

# Values from the online network for the sampled states.
q_online = torch.tensor([
    [0.70, 1.12, 0.20],
    [0.30, 0.10, 0.05],
    [0.40, 0.80, 1.60],
], requires_grad=True)

# Values from the frozen target network for the next states.
q_target_next = torch.tensor([
    [1.10, 1.55, 0.20],
    [0.40, 0.90, 0.30],
    [2.00, 1.30, 0.70],
])

q_taken = q_online.gather(1, actions[:, None]).squeeze(1)
next_best = q_target_next.max(dim=1).values
target = rewards + (1.0 - dones) * gamma * next_best

loss = ((target.detach() - q_taken) ** 2).mean()
loss.backward()

print("targets:", target.round(decimals=3).tolist())
print("q taken:", q_taken.detach().round(decimals=3).tolist())
print("loss:", round(loss.item(), 3))
print("gradient on q_online:", q_online.grad.round(decimals=3))

The terminal transition ignores the bootstrap term. The nonterminal transitions bootstrap from the target network, not from the changing online network. In a real DQN, the tensors come from neural network forward passes and an optimizer updates the online network weights.

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Use the Deep Q-Learning Replay Target Lab to predict which stabilizer matters in a sampled update.

Choose a case, inspect the replay-buffer sample, compare online and target-network next-action values, then commit before reveal. The lab unlocks the target calculation, TD error, approximate scalar update, and the caveat attached to the sampled transition.

Live Concept Demo

Explore Deep Q-Learning

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 4/5graduatecode-aligned

Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what Deep Q-Learning should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

Deep Q-learning turns Q-learning into neural value learning by training on replay-buffer samples, bootstrapping from a frozen target network, and watching for max-target overestimation.

Prediction open01 / Intuition

Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Deep Q-Learning should make visible.

Visual Inquiry

Make the image answer a mathematical question

Deep Q-learning turns Q-learning into neural value learning by training on replay-buffer samples, bootstrapping from a frozen target network, and watching for max-target overestimation.

4/4 stages readyLive demo connected

Visual cueWhich visible object should carry the first intuition?

Inspection depth2/4

Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Deep Q-Learning easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

paper · 2013Playing Atari with Deep Reinforcement LearningVolodymyr Mnih et al.

Introduces the DQN training recipe with a neural action-value function, experience replay, minibatch TD targets, and Atari evidence.

Open source

paper · 2015Human-level control through deep reinforcement learningVolodymyr Mnih et al.

Canonical DQN paper for replay memory, a separate target network, convolutional action-value learning from pixels, and the broader Atari evaluation.

Open source

course-notes · 2026Stanford CS234: Lecture 4, Model-Free Control and Function ApproximationStanford CS234

Graduate RL source for DQN instability causes, experience replay, fixed Q-targets, and target-network pseudocode.

Open source

paper · 2015Deep Reinforcement Learning with Double Q-learningHado van Hasselt, Arthur Guez, and David Silver

Source for the max-over-actions overestimation caveat and the Double DQN bridge.

Open source

Claim Review

Deep Q-learning turns Q-learning into neural value learning by training on replay-buffer samples, bootstrapping from a frozen target network, and watching for max-target overestimation.

Status1 substantive review recorded

Claims without a substantive review badge still need exact source-support review.

Sources4 references

mnih-2013-playing-atari-dqn, mnih-2015-human-level-control-dqn, stanford-cs234-dqn-2026, van-hasselt-2015-double-dqn

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedDeep Q-learning trains a neural action-value approximator with Q-learning targets computed from sampled replay-buffer transitions; a frozen target network supplies the bootstrap value, and max-over-actions targets can overestimate values, motivating Double DQN.Claim metadata: source checked

The sources support the DQN target y = r for terminal transitions and y = r + gamma max_a Q(s',a;theta-) otherwise, replay-buffer minibatch sampling, periodic target-network copies, and the overestimation risk from using a max over noisy value estimates.

Sources: Playing Atari with Deep Reinforcement Learning, Human-level control through deep reinforcement learning, Stanford CS234: Lecture 4, Model-Free Control and Function Approximation, Deep Reinforcement Learning with Double Q-learningTeaching lab only; excludes Atari preprocessing details, convolutional architecture tuning, reward clipping as a main topic, prioritized replay, dueling networks, distributional RL, Rainbow, and formal convergence proofs.A bounded review summary is present; still check caveats and exact reference scope.

Checked Mnih 2013 for replay memory, minibatch targets, the Q-network loss, and the experience-replay algorithm; checked Mnih 2015/Nature source material for replay memory and target-network DQN stabilization; checked Stanford CS234 Lecture 4 for the two instability causes and the fixed-target/replay pseudocode; checked van Hasselt et al. 2015 for DQN overestimation and the Double DQN decoupling caveat. GPT Pro critique remains pending because 127.0.0.1:51672 is unavailable.

Reviewer: codex-local-source-review; reviewed 2026-07-03

source-span-mnih-2013-playing-atari-dqn source-span-mnih-2015-human-level-control-dqn source-span-stanford-cs234-dqn-2026 source-span-van-hasselt-2015-double-dqn Equation 1 Equation 2 Runnable code 1 Interactive demo

Source support candidates

paper 2013Playing Atari with Deep Reinforcement Learning

Introduces the DQN training recipe with a neural action-value function, experience replay, minibatch TD targets, and Atari evidence.

paper 2015Human-level control through deep reinforcement learning

Canonical DQN paper for replay memory, a separate target network, convolutional action-value learning from pixels, and the broader Atari evaluation.

course-notes 2026Stanford CS234: Lecture 4, Model-Free Control and Function Approximation

Graduate RL source for DQN instability causes, experience replay, fixed Q-targets, and target-network pseudocode.

paper 2015Deep Reinforcement Learning with Double Q-learning

Source for the max-over-actions overestimation caveat and the Double DQN bridge.

Mechanism checks

Equation 1

Q(s,\cdot;\theta) = \left[ Q(s,a_1;\theta),\ldots,Q(s,a_m;\theta) \right],

Equation 2

(s_i,a_i,r_i,s'_i,d_i),

Runnable code 1import torch gamma = 0.90 # A minibatch of three replay-buffer transitions. actions = torch.t...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

Deep Q-learning turns Q-learning into neural value learning by training on replay-buffer samples, bootstrapping from a frozen target network, and watching for max-target overestimation.

Readiness0/3 checks ready

Predict

Before touching the demo, predict one visible change that should happen in Deep Q-Learning.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Your answer canvas

Local checks

Claim

A concrete answer is on the canvas.

Mechanism

The answer names why the claim should hold.

Bridge

It touches the page context or a neighboring idea.

Misconception check

Grounded research drawerClose

ConceptDeep Q-LearningReinforcement Learning

Runnable code comparisonDeep Q-Learning runnable code 1gamma = 0.90 Prediction before revealDeep Q-Learning interactive demoManipulate one control and predict the visible change.

Grounded room questionWhat is the smallest example that makes Deep Q-Learning click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptReinforcement Learning

Deep Q-Learning

Attached question

What is the smallest example that makes Deep Q-Learning click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action

Local action draft

This draft stays in this browser, attached to the selected learning item.

Draft noteNext action

No local draft saved.

Evidence to inspect

References to inspect: attached references on this page.
Definition, prerequisite, and contrast concept links
The equation or runnable code that makes the concept operational
One demo state that shows the invariant instead of a slogan

What would resolve this

The learner can state the mechanism in their own words
The learner can name the prerequisite that would repair confusion
The learner can predict how the mechanism changes under one perturbation

Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - Deep Q-Learning Selected item key: recorded for copy. Context: Reinforcement Learning Page anchor: recorded for copy. Open question: What is the smallest example that makes Deep Q-Learning click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context

concept/concept-notebook/reinforcement-learning/deep-q-learning
concept:reinforcement-learning/deep-q-learning

Learning Map

Before / Now / Try / Next

BeforeQ-Learning and SARSA (review)

NowIntuition -> Math -> Code -> Demo

TryManipulate one control and predict the visible change.

NextDouble Dqn (planned)

Intuitionready
Mathready
Codeready
Interactive Demoready

AI Companion

Ask beside the selected object

Deep Q-learning turns Q-learning into neural value learning by training on replay-buffer samples, bootstrapping from a frozen target network, and watching for max-target overestimation.

Your question

GoalComfortStyleStuck on

Context prompt

You are my AI learning companion for Continuous Function. Current context: Reinforcement Learning concept. Learning surface: Deep Q-Learning. What this page says: Deep Q-learning turns Q-learning into neural value learning by training on replay-buffer samples, bootstrapping from a frozen target network, and watching for max-target overestimation. Current section: Intuition, math, code, and interactive demo. Suggested next step: Manipulate one control and predict the visible change.. Learner goal: Understand the idea. Learner comfort level: New to this. Preferred explanation style: Visual first. Task: Explain the central idea in plain language, then restate it with the exact math objects from the page. Answer in a way that helps me learn: ask one clarifying question only if needed, use intuition before notation, and end with one thing I should try on the page.

Domain

Reinforcement Learning

reinforcement-learningdeep-rldqnreplay-buffertarget-networkvalue-function-approximation

Prerequisites

Q-Learning and SARSA (review)Function Approximation in RL (review)

Leads To

Double Dqn (planned)

Temporal-Difference Learning (review)MLPs and Universal Approximation (review)Policy Gradient (review)

Within this domain