Continuous Function

Foundation reference

KTO: Alignment from Binary Feedback via Human-Aware Losses

KTO trains a language model on outputs labeled only as desirable or undesirable, rather than on paired comparisons. Ethayarajh et al. (2024) argue that such labels are cheaper and more plentiful than preference pairs.

A newer notebook covers this concept. This older page keeps its summary, math and sources.

Concept 25 of 100Scaling & AlignmentStudy phase 7
Open the demoOpen the newer notebookAll concepts
Key equation
v(x,y)=λDσ ⁣(β(rθ(x,y)−z0))v(x,y)=\lambda_D\sigma\!\big(\beta(r_\theta(x,y)-z_0)\big)
Reading map and next steps

Study guide

Where to start

Concept 25 of 100KTOScaling & Alignment · Study phase 7: Alignment & RLHF
Question

Why is maximizing the likelihood of pairwise preferences, as DPO does, not the same as maximizing human utility?

Open the demoGo to the mathOpen the newer notebookNext concept
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.

Before you move a control in the demo, say what you expect to change.

CheckCheck against the math and the papers.

Compare what the demo shows with the key equation and with the papers cited below.

GeneralizeName what stays the same.

Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.

ContinueContinue in the newer notebook.

The newer notebook covers the same concept and has an interactive demo.

Why It Matters for Modern Models

  • KTO trains a language model on outputs labeled only as desirable or undesirable, rather than on paired comparisons. Ethayarajh et al. (2024) argue that such labels are cheaper and more plentiful than preference pairs.
  • Ethayarajh et al. (2024) report that KTO matches DPO even when up to 90% of the desirable examples are discarded, by reweighting the two classes with λ_D and λ_U.
  • KTO, DPO and related losses can be compared as different implied models of human utility, so the choice of loss acts as an inductive bias (Ethayarajh et al., 2024).
  • Because the sigmoid saturates, very easy, very hard or possibly mislabeled examples have little influence; the same property can make KTO underfit data that is hard but correct.
  • Compared with DPO, KTO shows that pairwise preferences are not required: in the reported experiments, per-example binary labels with a suitable loss matched or exceeded DPO.

Questions to Think About

  • What does the reference point z₀ do, and why does it stop the model from simply raising the likelihood of every desirable output?
  • Why can saturating gradients make KTO more robust to noisy labels, and when does the same saturation cause underfitting?
  • How much of alignment quality comes from the choice of objective rather than from the data, as KTO and binary classifier optimization (BCO) suggest?
  • When is binary feedback (like or dislike) enough, and when are paired comparisons worth their extra cost?

Interactive Demo

Core Math

If you prefer intuition first, try the demo above, then come back for the math.

Key Equation
v(x,y)=λDσ ⁣(β(rθ(x,y)−z0))v(x,y)=\lambda_D\sigma\!\big(\beta(r_\theta(x,y)-z_0)\big)

KTO aligns models using binary feedback (each output labelled desirable or undesirable) instead of pairwise comparisons. Its loss is built from a value function in the style of Kahneman and Tversky's prospect theory, and Ethayarajh et al. (2024) call losses of this kind human-aware losses (HALOs).

Implied reward (policy vs reference):

rθ(x,y)=log⁡πθ(y∣x)πref(y∣x)r_\theta(x,y) = \log \frac{\pi_\theta(y|x)}{\pi_{\text{ref}}(y|x)}

Reference point (baseline) as KL:

z0=KL ⁣(πθ(⋅∣x)∥πref(⋅∣x))z_0 = \text{KL}\!\left(\pi_\theta(\cdot|x) \| \pi_{\text{ref}}(\cdot|x)\right)

KTO loss with logistic value function:

LKTO(πθ,πref)=E(x,y)∼D[λy−v(x,y)]\mathcal{L}_{\text{KTO}}(\pi_\theta,\pi_{\text{ref}})=\mathbb{E}_{(x,y)\sim D}\big[\lambda_y - v(x,y)\big]

where

v(x,y)={λDσ ⁣(β(rθ(x,y)−z0))y∈desirableλUσ ⁣(β(z0−rθ(x,y)))y∈undesirablev(x,y)=\begin{cases} \lambda_D\sigma\!\big(\beta(r_\theta(x,y)-z_0)\big) & y \in \text{desirable}\\ \lambda_U\sigma\!\big(\beta(z_0-r_\theta(x,y))\big) & y \in \text{undesirable} \end{cases}

In practice z0z_0 is estimated from mismatched pairs in the batch and is not backpropagated through.

Writing z=rθ(x,y)−z0z = r_\theta(x,y) - z_0, the gradient carries a factor σ(βz)(1−σ(βz))\sigma(\beta z)(1-\sigma(\beta z)), which goes to zero when ∣z∣|z| is large. Examples the model already finds very easy or very hard therefore contribute little; learning concentrates on examples near the reference point.

Key Papers

KTO: Model Alignment as Prospect Theoretic Optimization

Ethayarajh et al.2024ICML
Read paper →

Binary Classifier Optimization for Large Language Model Alignment

Jung et al.2025ACL
Read paper →

Noise Contrastive Alignment of Language Models with Explicit Rewards

Chen et al.2024NeurIPS
Read paper →

Connections

Prerequisites

RLHFDPO

Leads to

Reward Hacking

Where to go next

A newer notebook covers KTO, with an interactive demo.

Continue in the newer notebook
Other connections (5)

Which limitations does it address, or leave to a later idea?

  • Preference pairs: a limitation of RLHF that KTO addresses. Open RLHF

    RLHF/DPO need paired preferences (A > B). KTO works with unpaired good/bad examples, using prospect theory's asymmetric value function.

  • No pairs needed: a limitation of DPO that KTO addresses. Open DPO

    KTO removes DPO's need for paired preferences: it learns from single outputs labelled desirable or undesirable, using a value function from Kahneman and Tversky's prospect theory (Ethayarajh et al., 2024).

Where else does the same idea appear?

  • Human-aware utility: the same technique appears in Learning Theory. Open Learning Theory

    KTO takes the shape of its loss from Kahneman and Tversky's prospect theory: outcomes are judged relative to a reference point, and losses weigh more than equal gains (Ethayarajh et al., 2024). It is an example of a training loss designed around a model of human judgment.

  • Shaping behaviour: the same technique appears in Embeddings. Open Embeddings

    Both move a model's behaviour towards what people prefer, by different routes: KTO trains the weights on outputs labelled desirable or undesirable, while representation steering leaves the weights alone and adds a direction to the hidden states at inference time. Whether training of this kind works by moving activations along such directions is an open question.

  • Preference-aware generation: the same technique appears in Decoding & Sampling. Open Decoding & Sampling

    KTO training shapes the model distribution toward human preferences. At inference, sampling strategies (temperature, top-p) further control this shaped distribution—both operate on the same output space with complementary goals.