4 / 4 Model-Free Methods

Cliff Notes

In the classic gridworld 'Cliff Walking' example, an agent training with SARSA tends to converge on a safer route that stays away from a cliff edge, while an agent training with Q-learning converges on the riskier, truly optimal path right along the edge (even though both use epsilon-greedy behavior during training). Why?

A

SARSA always uses a fully greedy policy at test time while Q-learning does not, which is unrelated to their update rules

B

SARSA is on-policy and bootstraps using the value of the action actually taken next (including occasional random exploratory steps), so it accounts for the risk of falling off during exploration, whereas Q-learning is off-policy and bootstraps using the max over next actions regardless of what the behavior policy would actually do

C

SARSA uses a smaller learning rate than Q-learning by definition, making it more conservative

D

Q-learning cannot represent negative rewards, so it never learns that the cliff is dangerous