Model-based Constrained Reinforcement Learning using Generalized Control Barrier Function

Model information can be used to predict future trajectories, so it has huge potential to avoid dangerous regions when applying reinforcement learning (RL) on real-world tasks, like autonomous driving. However, existing studies mostly use model-free constrained RL, which causes inevitable constraint violations. This paper proposes a model-based feasibility enhancement technique of constrained RL, which enhances the feasibility of policy using generalized control barrier function (GCBF) defined on the distance to constraint boundary. By using the model information, the policy can be optimized safely without violating actual safety constraints, and the sample efficiency is increased. The infeasibility in solving the constrained policy gradient is handled by an adaptive coefficient mechanism. We evaluate the proposed method in both simulations and real vehicle experiments in a complex autonomous driving collision avoidance task. The proposed method achieves up to four times fewer constraint violations and converges 3. 36 times faster than baseline constrained RL approaches.

Authors

• Haitong Ma
• Jianyu Chen 0002
• Shengbo Eben Li
• Ziyu Lin
• Yang Guan
• Yangang Ren
• Sifa Zheng

Keywords

• Adaptation models
• Uncertainty
• Reinforcement learning
• Predictive models
• Safety
• Trajectory
• Task analysis
• Control Barrier Functions
• Constrained Reinforcement Learning
• Sampling Efficiency
• Constraint Violation
• Real Vehicle
• Safety Constraints
• Application Of Reinforcement Learning
• System Dynamics
• Lagrange Multiplier
• Approximate Solution
• Inequality Constraints
• Autonomous Vehicles
• Optimal Policy
• Technical Solutions
• Constrained Optimization
• Safety Considerations
• Update Rule
• Form Of Constraints
• Distance Constraints
• Software Architecture
• Model-free Reinforcement Learning
• Conditional Value At Risk
• Value At Risk
• Policy Update
• Model-based Reinforcement Learning
• Model-based Optimization
• Reinforcement Learning Problem