Policy Gradient

Let $\pi_\theta(a \mid s)$ be a differentiable stochastic policy with parameters $\theta$. In an episodic MDP, a trajectory is

$$\tau = (S_0,A_0,R_1,\ldots,S_T).$$

Write $G(\tau)$ for the return from the trajectory, and define the objective

$$J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[G(\tau)].$$

The score-function identity starts from

$$\nabla_\theta J(\theta) = \nabla_\theta \sum_\tau p_\theta(\tau)G(\tau).$$

Multiplying and dividing by $p_\theta(\tau)$ gives

$$\nabla_\theta J(\theta) = \sum_\tau p_\theta(\tau)G(\tau)\nabla_\theta \log p_\theta(\tau) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ G(\tau)\nabla_\theta \log p_\theta(\tau) \right].$$

For an MDP whose start-state distribution and environment dynamics do not depend on $\theta$,

$$p_\theta(\tau) = \rho(S_0)\prod_{t=0}^{T-1} \pi_\theta(A_t \mid S_t)\,p(S_{t+1},R_{t+1}\mid S_t,A_t).$$

So the only differentiable terms in the trajectory probability are the policy probabilities:

$$\nabla_\theta \log p_\theta(\tau) = \sum_{t=0}^{T-1} \nabla_\theta \log \pi_\theta(A_t \mid S_t).$$

The basic REINFORCE estimator uses sampled returns:

$$g = \sum_{t=0}^{T-1} G_t \nabla_\theta \log \pi_\theta(A_t \mid S_t).$$

An action-independent baseline $b(S_t)$ can be subtracted without changing the expected gradient:

$$g = \sum_{t=0}^{T-1} \left(G_t - b(S_t)\right) \nabla_\theta \log \pi_\theta(A_t \mid S_t).$$

The baseline matters because the raw Monte Carlo estimator can be noisy. If $G_t$ is above the baseline, the sampled action is reinforced. If $G_t$ is below the baseline, the sampled action is suppressed. The baseline is not allowed to depend on the sampled action in a way that sneaks in extra action preference.

In the one-step two-action demo below, $G_t$ is just the observed reward $R$, and the sample update is

$$\theta \leftarrow \theta + \alpha (R-b)\nabla_\theta \log \pi_\theta(a).$$