The Baseline That Bites Back

A standard baseline choice in policy-gradient methods is the state-value function V^π(s). When you subtract V^π(s) from the return G_t, the resulting term G_t − V^π(s) is, in expectation, an unbiased estimate of which quantity?

The entropy bonus term added to the objective to encourage exploration.

The advantage function A^π(s,a) = Q^π(s,a) − V^π(s), measuring how much better action a is than the policy's average action at state s.

The TD(0) prediction error used only for fitting the value function, unrelated to the policy gradient.

The KL divergence between the current and previous policy, used to bound the update step size.