White RoomNEW

One Sweep Then Greedy

A two-state MDP, γ=0.9\gamma = 0.9, all transitions deterministic.

state action reward next state
s1s_1 aa 1 s1s_1
s1s_1 cc 0 s2s_2
s2s_2 bb 0 s1s_1
s2s_2 dd 2 s2s_2

The current policy π\pi takes aa in s1s_1 and bb in s2s_2. Policy evaluation is run to convergence, then one policy improvement step is taken.

Which action does the improved policy take in s2s_2?