White RoomNEW

Seven Over Four

Sutton and Barto's 4x4 gridworld. The two shaded corners are one terminal state, reward is 1-1 on every transition, γ=1\gamma = 1, and moves that would leave the grid keep the agent where it is. The policy is equiprobable random, so each of up, down, left, right has probability 0.25.

T 1 2 3
4 5 6 7
8 9 10 11
12 13 14 T

After the first sweep, v1(s)=1v_1(s) = -1 for all fourteen nonterminal states and v1(T)=0v_1(\text{T}) = 0.

Compute v2(1)v_2(1).