Confounding Robust Continuous Control via Automatic Reward Shaping

Reward shaping has been applied widely to accelerate Reinforcement Learning (RL) agents’ training. However, a principled way of designing effective reward shaping functions, especially for complex continuous control problems, remains largely under-explained. In this work, we propose to automatically learn a reward shaping function for continuous control problems from offline datasets, potentially contaminated by unobserved confounding variables. Specifically, our method builds upon the recently proposed causal Bellman equation to learn a tight upper bound on the optimal state values, which is then used as the potentials in the Potential- Based Reward Shaping (PBRS) framework. Our proposed reward shaping algorithm is tested with Soft-Actor-Critic (SAC) on multiple commonly used continuous control benchmarks and exhibits strong performance guarantees under unobserved confounders. More broadly, our work marks a solid first step towards confounding robust continuous control from a causal perspective. Code: https: //github. com/mateojuliani/confounding_robust_cont_ control Full version: https: //arxiv. org/abs/2602. 10305

Authors

• Mateo Juliani Columbia University
• Mingxuan Li Columbia University
• Elias Bareinboim Columbia University

Keywords

• CausalInference
• ReinforcementLearning
• UnobservedConfounder