Question 809

Score Function Sleight of Hand

REINFORCE turns ∇θE[R]\nabla_\theta E[R] into E[R⋅∇θlog⁡πθ(a∣s)]E[R \cdot \nabla_\theta \log \pi_\theta(a|s)] by rewriting ∇θπ(a∣s)\nabla_\theta \pi(a|s) as π(a∣s)⋅∇θlog⁡π(a∣s)\pi(a|s) \cdot \nabla_\theta \log \pi(a|s). What is this identity called, and why is it essential to the algorithm?

A

The chain rule for backpropagation through time; it propagates gradients across the time steps of a recurrent network.

B

The log-derivative (score function) trick; it lets the policy gradient be estimated purely from samples of πθ\pi_\theta, with no need to differentiate through the environment's dynamics.

C

The reparameterization trick; it moves the randomness into a deterministic function of an independent noise variable so gradients can flow through the sampling step.

D

The importance sampling trick; it reweights samples collected under an old policy to estimate expectations under a new policy.