Kahneman-Tversky Optimization

KTO turns binary desirable/undesirable labels into a reference-relative utility loss: push a labeled output's policy/reference log-ratio above or below a KL-derived baseline, with saturating gradients.

Advanced · Undergraduate mathematics · about 18 minutes

Reading map and next steps

Intuition

The idea in plain words, before the symbols.

Kahneman-Tversky Optimization (KTO) trains a policy from outputs labeled desirable or undesirable. We inspect one labeled output and compare its policy/reference log-ratio with a KL-derived baseline. The figure checks how the label changes the loss gradient and why the gradient shrinks far from the baseline. This notebook uses log probabilities and the KL divergence.

DPO is built on one quantity: the policy/reference log-ratio $\log(\pi_\theta(y\mid x)/\pi_{\mathrm{ref}}(y\mid x))$ .

That number says how much more the current policy likes a completion than the reference policy did. DPO uses it in pairs: the winner should beat the loser by more than the reference already made it beat the loser.

KTO asks a different question:

Can we use the same policy/reference ratio when the data is only thumbs-up or thumbs-down?

The answer is to compare one labeled output against a reference point. For a desirable output, KTO wants the output's policy/reference log-ratio to rise above the policy's average drift from the reference. For an undesirable output, KTO wants that log-ratio to fall below the same baseline.

The mechanism is:

KTO uses a label-dependent saturating utility of the margin $r_\theta(x,y)-z_0$ , where $r_\theta$ is the policy/reference log-ratio and $z_0$ is a KL-derived reference point.

The saturation matters. Once a desirable example is already far above the baseline, or an undesirable example is already far below it, the gradient shrinks. KTO focuses updates near the baseline, where $r_\theta(x,y)\approx z_0$ . That can protect against noisy labels, but it can also underfit examples that are hard yet important.

KTO was introduced by Ethayarajh et al. (2024), "KTO: Model Alignment as Prospect Theoretic Optimization". The name comes from prospect theory, and the borrowing is specific: KTO uses a reference point and a saturating value function, adapted from the value function of Tversky and Kahneman (1992), "Advances in prospect theory". It is not a claim that this utility curve is the true psychology of how people judge text.

Study prompts for this section: Intuition

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Mathematics

Definitions, assumptions and the derivation.

Equation 1 Equation 2 Equation 3

Setup

For a prompt $x$ , completion $y$ , trainable policy $\pi_\theta$ , and reference policy $\pi_{\mathrm{ref}}$ , define the KTO implied reward:

\begin{aligned} r_\theta(x,y) &= \log\frac{\pi_\theta(y\mid x)} {\pi_{\mathrm{ref}}(y\mid x)},\\ z_0 &= \mathrm{KL}(\pi_\theta(\cdot\mid x)\|\pi_{\mathrm{ref}}(\cdot\mid x)),\\ \delta &= r_\theta(x,y)-z_0,\\ v_D &= \lambda_D\,\sigma(\beta\delta),\\ v_U &= \lambda_U\,\sigma(-\beta\delta),\\ \mathcal L_y &= \lambda_y-v_y. \end{aligned}

With $z_0$ treated as fixed for the update and $s=\sigma(\beta\delta)$ , the label-dependent derivative through $r_\theta$ is

\begin{aligned} \frac{\partial \mathcal L_D}{\partial r_\theta} &= -\lambda_D\beta s(1-s),\\ \frac{\partial \mathcal L_U}{\partial r_\theta} &= \lambda_U\beta s(1-s). \end{aligned}

For an autoregressive language model, both log-probabilities are sequence log-probabilities: sums over the completion tokens.

Each training example has a binary label $\ell\in\{D,U\}$ , where $D$ means desirable and $U$ means undesirable.

The KL reference point

KTO does not compare $y$ to one rejected completion. It compares $y$ to the policy's average reference-relative reward:

z_0 = \mathrm{KL}(\pi_\theta(\cdot\mid x)\|\pi_{\mathrm{ref}}(\cdot\mid x)).

Equivalently, this is the current policy's expected implied reward at prompt $x$ :

z_0 = \mathbb E_{y'\sim\pi_\theta(\cdot\mid x)} [r_\theta(x,y')].

The important scalar is

\delta = r_\theta(x,y)-z_0.

A desirable output should have positive $\delta$ . An undesirable output should have negative $\delta$ .

In practical KTO training, $z_0$ is usually estimated from mismatched outputs in the same microbatch, clamped to be nonnegative, and treated as a stop-gradient quantity. It controls loss saturation; it is not itself the thing KTO tries to optimize through.

For a microbatch of size $m\ge 2$ , define $\rho_i$ as the policy/reference log-ratio of a shifted output $y_{j(i)}$ under prompt $x_i$ :

\rho_i = \log \frac{\pi_\theta(y_{j(i)}\mid x_i)} {\pi_{\mathrm{ref}}(y_{j(i)}\mid x_i)}.

A toy version of the paper's reference estimate is then

\hat z_0=\max(0,\operatorname{mean}_i \rho_i).

The point is not to reuse the labeled pair $(x_i,y_i)$ , because those completions were deliberately selected as good or bad and can have unrepresentative rewards.

Label-dependent value and loss

Tversky and Kahneman's value function is a power law with a steeper slope for losses. Ethayarajh et al. report that its exponent makes optimization numerically unstable, so they replace it with a logistic function. Like the power law, the logistic is concave for gains and convex for losses; unlike it, the logistic saturates. For a desirable example,

v_D = \lambda_D\,\sigma(\beta\delta).

For an undesirable example,

v_U = \lambda_U\,\sigma(-\beta\delta).

The default KTO loss subtracts this value from the label's class weight:

\mathcal L_y = \lambda_y - v_y.

Over the dataset, this is:

\mathcal L_{\mathrm{KTO}} = \mathbb E_{(x,y,\ell)\sim D} [\lambda_\ell-v_\ell(x,y)].

So the two label cases are:

\mathcal L_D = \lambda_D(1-\sigma(\beta\delta)),

and

\mathcal L_U = \lambda_U(1-\sigma(-\beta\delta)).

Gradient descent pushes desirable examples upward in reference-relative reward and undesirable examples downward. If $z_0$ is treated as fixed for the update, let $s=\sigma(\beta\delta)$ . Then

\frac{\partial \mathcal L_D}{\partial r_\theta} = -\lambda_D\beta s(1-s),

while

\frac{\partial \mathcal L_U}{\partial r_\theta} = \lambda_U\beta s(1-s).

Both gradients have their largest magnitude near $\delta=0$ , then saturate.

Here $\beta$ is KTO's value-function saturation parameter: the larger $\beta$ , the faster the value saturates. Ethayarajh et al. note that it plays a role similar to DPO's $\beta$ , but in KTO it is introduced directly to control saturation.

The weights $\lambda_D$ and $\lambda_U$ are class/utility weights, not a universal moral setting. They are often chosen to account for the ratio of desirable to undesirable examples. Increasing $\lambda_U$ emphasizes undesirable examples; increasing $\lambda_D$ emphasizes desirable examples.

How this differs from DPO

DPO uses a pairwise margin:

r_\theta(x,y_w)-r_\theta(x,y_\ell).

KTO uses a pointwise margin:

r_\theta(x,y)-z_0.

That is the bridge. DPO asks whether the winner beats the loser by enough. KTO asks whether a single labeled output sits on the correct side of a KL-derived baseline.

Limits of the mechanism

KTO is natural when feedback is already binary or when desirable and undesirable examples are imbalanced. If preference data is clean, consistent, and pairwise, DPO may be the better fit. If labels are noisy or intransitive, KTO's saturation can be useful because extreme examples receive small updates.

When KTO is trained from preference pairs, a common conversion is to treat the preferred response as desirable and the rejected response as undesirable. That is a simplifying assumption, not a law of the method. Naturally binary feedback is the cleaner setting for KTO.

The same property can fail. A hard desirable example with very negative implied reward may be ignored instead of learned. Ethayarajh et al. suggest a lower $\beta$ and more epochs to reduce this underfitting; that lowers the risk but does not remove it.

A natural next step is Process Reward Models: instead of labeling a whole completion as desirable or undesirable, ask what changes when feedback attaches to intermediate reasoning steps.

Study prompts for this section: Mathematics

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Code

The same calculation as runnable code, in the notation of the derivation.

Code example 1import numpy as np def sigmoid(x): return 1.0 / (1.0 + np.exp(-x)) # Batch of four labeled pr...python

This code mirrors the math: compute the policy/reference log-ratio, estimate a toy KL baseline, apply the label-dependent KTO loss, and inspect the gradient direction.

import numpy as np

def sigmoid(x):
    return 1.0 / (1.0 + np.exp(-x))

# Batch of four labeled prompt/completion examples.
# Shapes:
#   logp_theta, logp_ref, desirable are all (B,)
#   desirable=True means label D; False means label U.
logp_theta = np.array([-8.2, -5.1, -7.0, -6.4])
logp_ref   = np.array([-8.6, -5.0, -6.1, -6.9])
desirable  = np.array([ True, True, False, False])

beta = 0.2
lambda_D = 1.0
lambda_U = 1.0

# KTO implied reward: policy/reference sequence log-ratio.
r = logp_theta - logp_ref

# Toy microbatch estimate of the KL reference point.
# These are log-probabilities for mismatched pairs (x_i, y_j),
# not the original labeled pairs (x_i, y_i).
mismatch_logp_theta = np.array([-6.0, -7.2, -5.8, -8.0])
mismatch_logp_ref   = np.array([-6.3, -7.6, -6.0, -8.2])
mismatch_r = mismatch_logp_theta - mismatch_logp_ref
z0_hat = max(0.0, float(np.mean(mismatch_r)))

delta = r - z0_hat

value_D = lambda_D * sigmoid(beta * delta)
value_U = lambda_U * sigmoid(-beta * delta)

value = np.where(desirable, value_D, value_U)
lambda_y = np.where(desirable, lambda_D, lambda_U)
loss = lambda_y - value

# Stop-gradient z0: derivative is only through r.
s = sigmoid(beta * delta)
grad_D = -lambda_D * beta * s * (1.0 - s)
grad_U =  lambda_U * beta * s * (1.0 - s)
grad_r = np.where(desirable, grad_D, grad_U)

print("r_theta:", np.round(r, 3))
print("mismatched r for z0:", np.round(mismatch_r, 3))
print("z0_hat:", round(z0_hat, 3))
print("delta = r - z0:", np.round(delta, 3))
print("per-example KTO loss:", np.round(loss, 3))
print("d loss / d r:", np.round(grad_r, 3))
print("mean loss:", round(float(loss.mean()), 3))

assert np.all(grad_r[desirable] < 0)
assert np.all(grad_r[~desirable] > 0)

For desirable examples, the gradient is negative, so gradient descent increases $r_\theta$ . For undesirable examples, the gradient is positive, so gradient descent decreases $r_\theta$ . In both cases, the magnitude is largest near $r_\theta=z_0$ .

Study prompts for this section: Code

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Demo

Change an input, predict the result, and see which quantities respond.

ExploreChange one control at a time and name what responds.An optional observation guide follows the demo.

Explore Kahneman-Tversky Optimization

Loading the interactive demo…

Observation guide

Optional and ungraded. Opening a guide saves nothing.

Choose what to inspect in Kahneman-Tversky Optimization, then open its guide.

Use the demo to inspect one labeled example. Change the label, the implied reward $r_\theta$ , the KL baseline $z_0$ , $\beta$ , and the class weights. Watch the KTO loss and gradient flip direction between desirable and undesirable examples, while saturating away from the baseline.

Study prompts for this section: Demo

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Focus on

In focus

Concept: Kahneman-Tversky Optimization

What is the smallest example of Kahneman-Tversky Optimization you can work through by hand, and what does it show?

BeforeDirect Preference OptimizationNextReward Hacking: Overoptimizing Preference Proxies

DetailsAlignment

About this itemAlignment

3 sources citedChecking this browser

Reference id

concept:alignment/kto

Math3 equationsr_\theta(x,y) = \log\frac{\pi_\theta(y\mid x)} {\pi_{ref}(y\mid x)}; z_0 = KL(\pi_\theta(\cdo...CodeKahneman-Tversky Optimization code example 1assert np.all(grad_r[desirable] < 0)DemoKahneman-Tversky Optimization interactive demoChange one control at a time and name what responds.

After the first pass

Go deeper

Optional material for a second reading. Sources are always shown; the other panels open from their headings or from these links, and closing one keeps what you wrote or revealed.

What to look for in each sectionChoose a section and what you expect to see in it, then read short advice.

Section by section

Choose what to look for in each section

Choose what to look for (no demo yet)01 / Intuition

Illustration of pointwise desirable and undesirable feedback shaped by an asymmetric value curve.

What to look for

Start with the picture, metaphor, or geometric mechanism.

Choose first

Choose what you expect to see in this section of Kahneman-Tversky Optimization.

Questions to ask of the pictureChoose a part of the picture and what you expect, then read short advice.

Looking at the picture

Questions to ask of the picture

4 of 4 sections writtenNo live demo yet

Visual cueWhich part of the picture should you look at first?

Highlight size2/4

Question

Which part of the picture should you look at first?

Choose first

Pick the part of the picture you expect to explain Kahneman-Tversky Optimization.

Sources

References for this page

A reference shows where an idea comes from; it does not vouch for every step on this page.

paper · 2024KTO: Model Alignment as Prospect Theoretic OptimizationEthayarajh et al.

Why it is cited

ICML 2024. Defines KTO, which learns from binary desirable/undesirable labels instead of preference pairs, using a logistic value of the policy/reference log-ratio measured against a KL reference point.

Open source

paper · 1992Advances in prospect theory: Cumulative representation of uncertaintyTversky and Kahneman

Why it is cited

Journal of Risk and Uncertainty 5(4). Gives the reference-dependent, loss-averse value function that KTO adapts; KTO replaces its power-law form with a logistic function.

Open source

paper · 2023Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafailov et al.

Why it is cited

NeurIPS 2023. The pairwise method KTO is compared with; both use the policy/reference log-ratio as an implicit reward.

Open source

What each claim rests on1 claim, each with its sources, where it appears on this page, and its caveats.

Claims and sources

KTO turns binary desirable/undesirable labels into a reference-relative utility loss: push a labeled output's policy/reference log-ratio above or below a KL-derived baseline, with saturating gradients.

Checked against the cited passages

The checks were made by this site, not by an independent reviewer.

KTO trains from binary desirable/undesirable feedback using a label-dependent logistic value of the margin r_theta(x,y)-z0, where r_theta is a policy/reference log-ratio and z0 is KL-derived; desirable updates increase the margin, undesirable updates decrease it, and beta controls saturation.

Sources

KTO: Model Alignment as Prospect Theoretic OptimizationICML 2024. Defines KTO, which learns from binary desirable/undesirable labels instead of preference pairs, using a logistic value of the policy/ref...

What the sources support

Ethayarajh et al. define KTO with binary labels, the implied reward log(pi_theta/pi_ref), a KL reference point z0 and logistic values scaled by beta. The math here derives the gradient signs with z0 held fixed.

Where to see it on this page

KTO: Model Alignment as Prospect Theoretic Optimization Equation 1 Equation 2 Code example 1 Demo

Equation 1

\begin{aligned} r_\theta(x,y) &= \log\frac{\pi_\theta(y\mid x)} {\pi_{\mathrm{ref}}(y\mid x)},\\ z_0 &= \mathrm{KL}(\pi_\theta(\cdot\mid x)\|\pi_{\mathrm{ref}}(\cdot\mid x)),\\ \delta &= r_\theta(x,y)-z_0,\\ v_D &= \lambda_D\,\sigma(\beta\delta),\\ v_U &= \lambda_U\,\sigma(-\beta\delta),\\ \mathcal L_y &= \lambda_y-v_y. \end{aligned}

Equation 2

\begin{aligned} \frac{\partial \mathcal L_D}{\partial r_\theta} &= -\lambda_D\beta s(1-s),\\ \frac{\partial \mathcal L_U}{\partial r_\theta} &= \lambda_U\beta s(1-s). \end{aligned}

Caveat

Covers the KTO objective and its gradients only. It does not claim that KTO outperforms DPO, that prospect theory describes how people judge text, that every library estimates z0 this way, or that KTO preven...

StatusChecked against the cited passages.

Ethayarajh et al. define KTO for binary desirable/undesirable labels, with r_theta = log(pi_theta/pi_ref), a KL reference point z0, logistic values for each label and beta controlling saturation. The first two equations on this page state the objective and the signs of its derivatives with z0 held fixed; the code and demo show the same update directions.

Explain it without the pageWrite your own explanation before asking for feedback. Your draft and any hints stay while this page is open.

Practice · Kahneman-Tversky Optimization

Try the idea in your own words

Concept · Selected for practice

Kahneman-Tversky Optimization

What it rests on: Sources: KTO: Model Alignment as Prospect Theoretic Optimization; Advances in prospect theory: Cumulative representation of uncertainty; Direct Pre...

Context and links

Alignment

Choose a task

Explain the mechanism

For Kahneman-Tversky Optimization: What is the smallest example of Kahneman-Tversky Optimization you can work through by hand, and what does it show? Explain your answer, including what changes, why, and which assumption matters.

Your answerNo answer yet

A rough first thought is enough. Your draft stays when you change tasks.

Your draft stays on this page and is cleared when you leave or reload.

A little help · Explain

Open one hint at a time. These are suggestions, not your answer or a grade.

0 of 3 hints shown for this question.

Clearing your answer keeps this help record. Outside help is your own report; this page cannot check it.

Where am I stuck? (optional)

Your own description, not an automatic diagnosis

Choose one, or leave this unspecified. Select it again to clear it.

Take your draft to a feedback conversation

No AI feedback runs here. You can copy a prompt to use elsewhere; nothing is sent automatically. Review the text before sharing, and leave out private information.

Write an attempt before copying a feedback prompt.

A draft, or an AI reply to it, is not a test of what you have learned. To check that, try a different case later without help.

Ask about this pagePick one item on this page and copy a prompt about it into an AI assistant you already use. Notes you write here stay in this browser.

Ask about this pageClose

ConceptKahneman-Tversky OptimizationSources: KTO: Model Alignment as Prospect Theoretic Optimization; Advances in prospect theory: Cumulative representation of uncertainty; Direct Pre...assert np.all(grad_r[desirable] < 0)

Your question

Choose what your question is about

Pick the idea, equation, source, code, claim, misconception or demo state you want to ask about. The prompt you copy includes it, so the answer can stay on that item.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptAlignment

Kahneman-Tversky Optimization

Starting question

What is the smallest example of Kahneman-Tversky Optimization you can work through by hand, and what does it show?

SourcesCheck the 3 cited sources listed under Sources.Notes can be saved for this item

Points of view on this item

Fixed prompts generated from this item. They are not comments from people or independent reviews.

Learner’s first stepAsk what would make "Kahneman-Tversky Optimization" feel predictable rather than familiar.

Assumption

The 3 cited sources must support this exact item, not just the surrounding topic.

Source-checking summary

Connect the definition to one equation, piece of code or demo before widening the discussion.

Proposed experiment

Change one input, then check that the same relationship holds in the mathematics, the code and the demo.

Next action

You can state the mechanism in your own words

Evidence4 checks

ObservationChecking for a saved observation

ActionReady for one action

PromptLearner prompt ready to copy

Go to this item

01ObservationChecking what this browser has saved

02EvidenceChecking for a saved observation

03SourcesCheck the 3 cited sources listed under Sources.

04Next stepSave one next action

Local action draftNo local draft saved yetOpen when you are ready to write one next action

Local action draft

This draft stays in this browser and is attached to this item only.

Draft noteNext action

No local draft saved.

Evidence to inspect

What the 3 cited sources say about this exact item
The definition, its prerequisites, and a contrasting concept
The equation or code that makes the concept concrete
One demo state that shows what stays the same as the inputs change

What would resolve this

You can state the mechanism in your own words
You can name the prerequisite that would clear up the confusion
You can predict how the result changes when one input changes

Reference id

concept/concept-notebook/alignment/kto
concept:alignment/kto

Return to reading

Reading map

In this concept

Change one control at a time and name what responds.

Try the demo

Continue the curriculum

Part VII · unit 3 of 7

Reward Hacking: Overoptimizing Preference Proxies

Connections and sources

Domain

Alignment

alignment
kto
binary-feedback
halos

Prerequisites

Direct Preference Optimization

Leads to

Process Reward Models: Step-Level Verifiers for Reasoning

Study aid

Ask an assistant about this notebook

Compose a question with this notebook as context, then copy it into an AI assistant you already use.

Compose a question

Your question

GoalComfortStyleStuck on

Copy-only tool: this panel composes a prompt from this page and your choices above, for an AI assistant you already use. Nothing you enter here is sent from this page.