Bandits isolate the exploration problem: each pull reveals only one arm's reward, so the learner must trade sample-mean exploitation against information-gathering and regret.

Bandits

Exploration and exploitation is the horizon tradeoff: take known reward now, or spend pulls reducing uncertainty so future actions regret less.

Exploration and Exploitation

Bellman equations make value recursive: a state's value must equal immediate reward plus discounted value flowing back from the next states.

Bellman Equations

Value iteration turns the Bellman optimality backup into a known-model planning algorithm: sweep old values through every state until a greedy policy stabilizes.

Dynamic Programming and Value Iteration

An MDP is the RL contract for one-step prediction: current state plus action gives a next-state and reward law, while a policy mixes action kernels into behavior.

Markov Decision Process Formalism

Policy iteration alternates fixed-policy value evaluation with greedy policy improvement until the policy stops changing.

Policy Iteration

Q-learning and SARSA both update the action just taken; SARSA bootstraps from the next action the behavior policy sampled, while Q-learning bootstraps from the largest next action value in the current table.

Q-Learning and SARSA

Policy gradients update a stochastic policy directly: a sampled action becomes more or less likely according to its return relative to a baseline, through grad log pi.

Policy Gradient

RLHF can be read as policy optimization over sampled completions, learned proxy rewards, and KL anchoring, but the language-model setting breaks several textbook MDP assumptions.

RLHF as Reinforcement Learning

Deep Q-learning turns Q-learning into neural value learning by training on replay-buffer samples, bootstrapping from a frozen target network, and watching for max-target overestimation.

Deep Q-Learning

Function approximation replaces a Q-table with features and weights, so one TD update can generalize to nearby state-action pairs and can become unstable when bootstrapping, off-policy data, and approximation combine.

Function Approximation in RL

Monte Carlo RL evaluates a fixed policy by averaging return-to-go values from complete sampled episodes, so no transition model is needed and small visit counts can be noisy.

Monte Carlo Reinforcement Learning

Temporal-difference learning updates a state value after one sampled transition by comparing the old value with a reward-plus-next-value target.

Temporal-Difference Learning

How agents choose actions over time: states, actions, rewards, transition dynamics, value functions, exploration, and the policy-optimization ideas behind modern RL and preference learning.

Reinforcement Learning

Start here, then follow the prerequisites forward.

Browse the territory.

Notebooks that aren't published yet.