White RoomNEW

One Episode, Two Updates

Tabular prediction with α=0.1\alpha = 0.1, γ=1\gamma = 1, and every state initialised to V=0.5V = 0.5. A single episode is observed:

Ar=0Br=0Cr=1terminalA \xrightarrow{,r=0,} B \xrightarrow{,r=0,} C \xrightarrow{,r=1,} \text{terminal}

Apply constant-α\alpha Monte Carlo, V(St)V(St)+α[GtV(St)]V(S_t) \leftarrow V(S_t) + \alpha[G_t - V(S_t)], and TD(0), V(St)V(St)+α[Rt+1+γV(St+1)V(St)]V(S_t) \leftarrow V(S_t) + \alpha[R_{t+1} + \gamma V(S_{t+1}) - V(S_t)]. What is V(A)V(A) after this one episode under each method?