Farming The Shaping Reward

A PPO agent must reach a goal; the environment pays $+1$ on arrival and nothing otherwise. Learning stalls, so the team adds a shaping reward: $+0.1$ every step in which the distance to the goal decreased.

The agent stops reaching the goal at all. It learns to move toward the goal, back away, and repeat, collecting $+0.1$ on every second step forever.

Which change keeps a dense learning signal and provably leaves the optimal policy of the original task unchanged?

Keep the shaping and add an entropy bonus, so the policy does not collapse onto the oscillation.

Pay the shaping reward as a potential difference, $F(s,a,s') = \gamma \Phi(s') - \Phi(s)$ with $\Phi(s) = -\text{distance}(s, \text{goal})$ , so the shaping an agent can collect telescopes to a constant fixed by its start state.

Also pay $-0.1$ whenever the distance increases, so the shaping is sign-symmetric and any round trip nets out to zero.

Cut the shaping to $+0.05$ and cap the shaping total per episode, so the goal reward always dominates.