Seven Over Four

Sutton and Barto's 4x4 gridworld. The two shaded corners are one terminal state, reward is −1-1 on every transition, γ=1\gamma = 1, and moves that would leave the grid keep the agent where it is. The policy is equiprobable random, so each of up, down, left, right has probability 0.25.

T 1 2 3
4 5 6 7
8 9 10 11
12 13 14 T

After the first sweep, v1(s)=−1v_1(s) = -1 for all fourteen nonterminal states and v1(T)=0v_1(\text{T}) = 0.

Compute v2(1)v_2(1).

A

−2.25-2.25

B

−1.75-1.75

C

−2.0-2.0

D

−1.0-1.0