Gradient-based Regularization for Action Smoothness in Robotic Control with Reinforcement Learning

Deep Reinforcement Learning (DRL) has achieved remarkable success, ranging from complex computer games to real-world applications, showing the potential for intelligent agents capable of learning in dynamic environments. However, its application in real-world scenarios presents challenges, including the jerky problem, in which jerky trajectories not only compromise system safety but also increase power consumption and shorten the service life of robotic and autonomous systems. To address jerky actions, a method called conditioning for action policy smoothness (CAPS) was proposed by adding regularization terms to reduce the action changes. This paper further proposes a novel method, named Gradient-based CAPS (Grad-CAPS), that modifies CAPS by reducing the difference in the gradient of action and then uses displacement normalization to enable the agent to adapt to invariant action scales. Consequently, our method effectively reduces zigzagging action sequences while enhancing policy expressiveness and the adaptability of our method across diverse scenarios and environments. In the experiments, we integrated Grad-CAPS with different reinforcement learning algorithms and evaluated its performance on various robotic-related tasks in DeepMind Control Suite and OpenAI Gym environments. The results demonstrate that Grad-CAPS effectively improves performance while maintaining a comparable level of smoothness compared to CAPS and Vanilla agents.

Authors

• I Lee
• Hoang-Giang Cao
• Cong-Tinh Dao
• Yu-Cheng Chen
• I-Chen Wu

Keywords

• Video games
• Power demand
• Autonomous systems
• Deep reinforcement learning
• Distance measurement
• Trajectory
• Safety
• Intelligent agents
• Intelligent robots
• Robot Control
• Autonomic System
• Sequence Of Actions
• Diverse Environments
• Robotic System
• Regularization Term
• Vanilla
• Reinforcement Learning Algorithm
• Gradient Difference
• Changes In Activity
• Differences In Activity
• Complex Environment
• Loss Of Expression
• Derivative Of Function
• Control Task
• Square Wave
• Lipschitz Continuous
• Robotic Arm
• Smooth Trajectory
• Average Reward
• Temporal Smoothing
• Performance Of Agents
• Temporal Loss
• Smoothing Value
• Humanoid
• Temporal Terms
• Regularization Method
• Spatial Smoothing