Safe Offline Reinforcement Learning using Diffusion Policies

DiffusionmodelshaveshowngreatpromiseforofflineRLbycapturing complex data distributions, yet their standard formulations lack explicit safety mechanisms. We propose Safe Diffusion Q-learning, which extends Diffusion-QL by integrating a cost critic and a direct penalty term into the diffusion policy objective to enforce constraint satisfaction during action generation. Evaluated on the DSRL benchmark, our method achieves near-zero constraint violation on challenging BulletSafetyGym and SafetyGym tasks while maintainingcompetitiverewardperformance. Theseresultsdemonstrate that expressive diffusion policies can be robustly constrained, enabling their use in safety-critical offline RL.

Authors

• Ankita Kushwaha IIIT Hyderabad
• Kiran Ravish IIIT Hyderabad
• Preeti IIIT Hyderabad
• Pawan Kumar IIIT Hyderabad
• Anuj Mahajan Meta

Keywords

• Safe Offline RL
• Diffusion Policies
• Q-Learnig
• Single Agent