PPO: Proximal Policy Optimization

Why It Matters for Modern Models

Proximal policy optimization (PPO) updates a policy with a clipped objective based on data from its previous version. PPO was the policy-update step in InstructGPT (Ouyang et al., 2022); later pipelines often use direct methods such as DPO instead.
Clipping is a practical stand-in for a trust region: it removes the incentive to push an action's probability ratio past 1 ± ε but sets no hard bound on the update. In RLHF, a separate KL penalty to the starting model discourages the policy from drifting far over the whole of training (Ouyang et al., 2022).
GAE's λ trades bias against variance in the advantage estimates, and noisy advantages are a common cause of unstable policy-gradient training (Schulman et al., 2016).

Questions to Think About

Why can the probability ratio be read as reweighting old experience by how much more or less likely each action has become?
How does noise in a learned reward model pass into the advantage estimates, and what does it do to PPO updates in RLHF?

Core Math

Key Equation

L^{CLIP}(\theta) = \mathbb{E}_t\left[ \min\left( \rho_t \hat{A}_t,\ \text{clip}(\rho_t, 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right]

PPO updates a policy by maximizing a clipped surrogate objective on data collected by the previous policy $\pi_{\theta_{\text{old}}}$ :

L^{CLIP}(\theta) = \mathbb{E}_t\left[ \min\left( \rho_t(\theta) \hat{A}_t,\ \text{clip}(\rho_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right]

where the probability ratio is

\rho_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}

and $\hat A_t$ estimates how much better action $a_t$ was than average in state $s_t$ . (Schulman et al. write the ratio as $r_t(\theta)$ ; it is called $\rho_t$ here to avoid a clash with the reward.)

Generalized advantage estimation (GAE):

\hat{A}_t = \sum_{l=0}^{\infty} (\gamma\lambda)^l \delta_{t+l}, \quad \delta_t = R_t + \gamma V(s_{t+1}) - V(s_t)

with reward $R_t$ , discount $\gamma$ and learned value estimate $V$ ; $\lambda$ trades bias against variance (Schulman et al., 2016).

Taking the minimum with the clipped term removes the incentive to push $\rho_t$ outside $[1-\epsilon, 1+\epsilon]$ when doing so would raise the objective, and keeps the unclipped term when the change would lower it, so $L^{CLIP}$ is a pessimistic lower bound on the unclipped surrogate (Schulman et al., 2017, Section 3). It is not a hard limit on the update: nothing projects the ratio back into the interval, and the objective by itself guarantees neither stable training nor protection against forgetting. It gives much of the practical benefit of a trust-region method while using only first-order optimization.

Group-relative policy optimization (GRPO) keeps PPO's clipped ratio but drops the learned value function, and with it GAE. It samples a group of outputs for the same prompt and weights each one by its reward minus the group's mean reward, divided by the group's standard deviation (Shao et al., 2024). Work through one group-relative update in the RLHF notebook.