The Clip That Keeps You Honest

PPO's clipped surrogate objective is $L^{CLIP}(\theta) = E[\min(r_t(\theta)\hat{A}$ , where $r_t(\theta) = \pi$ \theta(a_t|s_t) / \pi{\theta_{old}}(a_t|s_t) $r_{t} (θ) = π_{θ} (a_{t} ∣ s_{t}) / π_{θ_{o l d}} (a_{t} ∣ s_{t})$ . What effect does the $\mathrm{clip}(...)$ term achieve?

It clips the reward signal $r_t$ to $[1-\epsilon, 1+\epsilon]$ before computing returns, acting as reward normalization.

It removes the incentive to push $r_t(\theta)$ far outside $[1-\epsilon, 1+\epsilon]$ when doing so would further increase the objective, discouraging destructively large policy updates while still allowing moves that decrease it.

It guarantees the KL divergence between $\pi_\theta$ and $\pi_{\theta_{old}}$ never exceeds $\epsilon$ , exactly like TRPO's hard constraint.

It rescales the advantage estimates $\hat{A}_t$ to lie strictly within $[-\epsilon, \epsilon]$ before the policy gradient step.