Five Now Or Twenty Later

From the start state of an episodic MDP an agent can commit to one of two policies. Rewards arrive at the steps shown and are 0 everywhere else.

policy	reward stream from $t$
A	$R_{t+1} = 5$ , then 0 forever
B	0 for four steps, then $R_{t+5} = 20$ , then 0 forever

Using $G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \cdots$ , what are the two returns at $\gamma = 0.9$ , and which policy is optimal?

$G_A = 5$ and $G_B = 13.12$ , but A is optimal, because any $\gamma < 1$ makes the agent myopic and myopic agents maximise the immediate reward.

$G_A = 5$ and $G_B = 0.9^4 \cdot 20 = 13.12$ , so B is optimal at $\gamma = 0.9$ ; at $\gamma = 0.5$ the same MDP gives $G_B = 1.25$ and A becomes optimal.

$G_A = 5$ and $G_B = 20$ , so B is optimal at any $\gamma$ : discounting only keeps infinite sums finite and does not change a comparison between two finite reward streams.

$G_A = 4.5$ and $G_B = 0.9^5 \cdot 20 = 11.81$ , so B is optimal, since every reward in the return is discounted at least once.