Accelerating Action-Robust Deep Deterministic Policy Gradient via Parallel Optimization

The Noisy Action Robust MDP (NR-MDP) framework is effective for training robust policies but suffers from significant computational inefficiency due to the iterative, alternating training of a protagonist and an adversary. While reducing the frequency of adversary updates can shorten training time, it typically results in a degradation of robust performance. To address this trade-off, we propose Efficient Action Robust DDPG (EAR-DDPG), which eliminates the need for explicit adversarial policy learning by generating adversarial actions on-the-fly. We show that the approximately optimal adversarial action lies at a vertex of the hypercube. This insight allows us to replace conventional iterative optimization with instantaneous parallel computation over these vertex candidates.

Authors

• SeongIn Kim University of Tsukuba
• Takeshi Shibuya University of Tsukuba

Keywords

• Robust reinforcement learning
• Adversarial learning
• Parallel computing