White RoomNEW

Same Transition, Two Updates

An agent takes action aa in state ss, receives r=2r = 2, and lands in ss'. Current estimates:

Q(s,a)=4.0,Q(s,up)=3.0,Q(s,down)=1.0,Q(s,right)=7.0,Q(s,left)=0.0Q(s,a) = 4.0, \qquad Q(s', \text{up}) = 3.0, \quad Q(s', \text{down}) = -1.0, \quad Q(s', \text{right}) = 7.0, \quad Q(s', \text{left}) = 0.0

The ε\varepsilon-greedy behaviour policy explores at ss' and selects down. With α=0.5\alpha = 0.5 and γ=0.9\gamma = 0.9, what is Q(s,a)Q(s,a) after this transition under Q-learning and under Sarsa?