Domain
Alignment
Training language models from human feedback: reward models fit to comparisons, KL-regularized fine-tuning, direct preference methods, step-level verifiers, and what goes wrong when a learned proxy is optimized too hard.
Preview the recommended route
Start here. Make one prediction, then note what carries over to the next notebook.
π_θpreferencerewardKL guardrailLearnerWhich notebook should I read first?
Open the first notebookRead the first notebook, make one prediction in its demo, then go on to the next.
Recommended route
Start here, then follow the prerequisites forward.
Read these in order: each notebook comes after the notebooks in this domain that it builds on, and the label beside it gives its level. A prerequisite with no page here yet is marked as outside reading. A notebook is left out of the route only when a prerequisite has neither a page here nor a named outside reading; the full list below marks it.
- 01RLHF: Reward Modeling + KL-Regularized Policy Optimization
RLHF trains a reward model from pairwise preferences, then reweights a reference policy toward high learned reward while a KL penalty limits distribution shift.
Advanced20 mincodedemoafter Maximum Likelihood, Cross-Entropy, KL Divergence (Relative Entropy) - 02Reward Hacking: Overoptimizing Preference Proxies
When an imperfect learned reward is optimized beyond the range where it was validated, probability shifts toward the reward model's errors; KL penalties, ensembles, lower-confidence-bound scores and monitoring slow this down but do not make the proxy true.
Advanced22 mincodedemoafter RLHF: Reward Modeling + KL-Regularized Policy OptimizationReward Hacking builds directly on RLHF.
- 03Direct Preference Optimization
DPO turns pairwise preferences into binary cross-entropy on reference-relative log odds, using the KL-regularized RLHF optimum to make the policy itself an implicit reward model.
Advanced18 mincodedemoafter Cross-Entropy, KL Divergence (Relative Entropy), RLHF: Reward Modeling + KL-Regularized Policy Optimization - 04Kahneman-Tversky Optimization
KTO turns binary desirable/undesirable labels into a reference-relative utility loss: push a labeled output's policy/reference log-ratio above or below a KL-derived baseline, with saturating gradients.
Advanced18 mincodedemoafter Direct Preference OptimizationKahneman-Tversky Optimization builds directly on Direct Preference Optimization.
- 05Process Reward Models: Step-Level Verifiers for Reasoning
A process reward model scores intermediate reasoning steps rather than only the final answer. These learned scores can guide reranking and search, but remain proxies for correctness.
Advanced15 mincodedemoafter RLHF: Reward Modeling + KL-Regularized Policy Optimization, Reward Hacking: Overoptimizing Preference Proxies, Cross-Entropy
All notebooks
Every published notebook in this domain.
RLHF: Reward Modeling + KL-Regularized Policy Optimization
RLHF trains a reward model from pairwise preferences, then reweights a reference policy toward high learned reward while a KL penalty limits distribution shift.
Reward Hacking: Overoptimizing Preference Proxies
When an imperfect learned reward is optimized beyond the range where it was validated, probability shifts toward the reward model's errors; KL penalties, ensembles, lower-confidence-bound scores and monitoring slow this down but do not make the proxy true.
Direct Preference Optimization
DPO turns pairwise preferences into binary cross-entropy on reference-relative log odds, using the KL-regularized RLHF optimum to make the policy itself an implicit reward model.
Kahneman-Tversky Optimization
KTO turns binary desirable/undesirable labels into a reference-relative utility loss: push a labeled output's policy/reference log-ratio above or below a KL-derived baseline, with saturating gradients.
Process Reward Models: Step-Level Verifiers for Reasoning
A process reward model scores intermediate reasoning steps rather than only the final answer. These learned scores can guide reranking and search, but remain proxies for correctness.