Domain Neighborhood
Alignment
How we shape model behavior: preference learning, reward modeling, KL-regularized fine-tuning, and the failure modes that appear when you optimize the wrong thing.
Start here. Predict once, then carry the invariant forward.
π_θpreferencerewardKL guardrailLearner lensWhat makes this domain feel navigable?
Take this lensStabilize the first mechanism, make one prediction, then move one node forward.
Recommended Route
Start here, then follow the prerequisites forward.
This sequence is ordered for learning rather than inventory. Published notebooks with an unavailable prerequisite—or a same-domain route step that depends on one—are labeled in the full inventory instead of being presented as ready steps.
- 01RLHF: Reward Modeling + KL-Regularized Policy Optimization
RLHF trains a reward model from pairwise preferences, then reweights a reference policy toward high learned reward while a KL penalty limits distribution shift.
20 mincodedemoafter Maximum Likelihood, Cross-Entropy, KL Divergence (Relative Entropy)Check Maximum Likelihood first if the symbols feel slippery.
- 02Reward Hacking: Overoptimizing Preference Proxies
When an imperfect preference proxy is optimized past its validation regime, policy mass shifts toward reward-model errors; KL, ensembles, LCBs, and monitoring slow this down but do not make the proxy true.
22 mincodedemoafter RLHF: Reward Modeling + KL-Regularized Policy OptimizationWhy this follows: Reward Hacking: Overoptimizing Preference Proxies uses RLHF: Reward Modeling + KL-Regularized Policy Optimization directly.
- 03Direct Preference Optimization
DPO turns pairwise preferences into binary cross-entropy on reference-relative log odds, using the KL-regularized RLHF optimum to make the policy itself an implicit reward model.
18 mincodedemoafter Cross-Entropy, KL Divergence (Relative Entropy), RLHF: Reward Modeling + KL-Regularized Policy OptimizationWhy this follows: both pages keep the alignment thread active.
- 04Kahneman-Tversky Optimization
KTO turns binary desirable/undesirable labels into a reference-relative utility loss: push a labeled output's policy/reference log-ratio above or below a KL-derived baseline, with saturating gradients.
18 mincodedemoafter Direct Preference OptimizationWhy this follows: Kahneman-Tversky Optimization uses Direct Preference Optimization directly.
- 05Process Reward Models: Step-Level Verifiers for Reasoning
A process reward model scores intermediate reasoning steps instead of only terminal answers, giving denser verifier feedback for reranking and search while remaining a learned proxy.
24 mincodedemoafter RLHF: Reward Modeling + KL-Regularized Policy Optimization, Reward Hacking: Overoptimizing Preference Proxies, Cross-EntropyWhy this follows: both pages keep the alignment thread active.
All Published Notebooks
Browse the territory.
RLHF: Reward Modeling + KL-Regularized Policy Optimization
RLHF trains a reward model from pairwise preferences, then reweights a reference policy toward high learned reward while a KL penalty limits distribution shift.
Reward Hacking: Overoptimizing Preference Proxies
When an imperfect preference proxy is optimized past its validation regime, policy mass shifts toward reward-model errors; KL, ensembles, LCBs, and monitoring slow this down but do not make the proxy true.
Direct Preference Optimization
DPO turns pairwise preferences into binary cross-entropy on reference-relative log odds, using the KL-regularized RLHF optimum to make the policy itself an implicit reward model.
Kahneman-Tversky Optimization
KTO turns binary desirable/undesirable labels into a reference-relative utility loss: push a labeled output's policy/reference log-ratio above or below a KL-derived baseline, with saturating gradients.
Process Reward Models: Step-Level Verifiers for Reasoning
A process reward model scores intermediate reasoning steps instead of only terminal answers, giving denser verifier feedback for reranking and search while remaining a learned proxy.
Advanced Bridges