4 / 5 Policy Gradient Methods

Lambda's Balancing Act

In an actor-critic setup, the critic estimates V(s)V(s), and GAE computes A^tGAE(γ,λ)=∑l(γλ)lδt+l\hat{A}t^{GAE(\gamma,\lambda)} = \sum_l (\gamma\lambda)^l \delta{t+l}, where δt=rt+γV(st+1)−V(st)\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t). What happens to the estimator's bias and variance as λ→0\lambda \to 0 versus λ→1\lambda \to 1?

A

λ→1\lambda \to 1 removes both bias and variance simultaneously because it averages over strictly more terms.

B

λ→0\lambda \to 0 gives the low-variance, high-bias one-step TD advantage (just δt\delta_t); λ→1\lambda \to 1 gives the high-variance, low-bias Monte Carlo-style advantage estimate as the residuals telescope into the full return.

C

λ→0\lambda \to 0 gives the unbiased Monte Carlo return; λ→1\lambda \to 1 collapses to a single TD residual with lower variance.

D

λ\lambda only rescales the discount factor γ\gamma, so it has no independent effect on bias or variance.