Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping

Some well-known "bugs" in reward shaping procedures are shown to arise from non-potential-based rewards, and methods are given for constructing shaping potentials corresponding to distance-based and subgoal-based heuristics that can lead to substantial reductions in learning time.