Diffusion Policies with Value-Conditional Optimization for Offline Reinforcement Learning

In offline reinforcement learning, value overestimation caused by out-of-distribution (OOD) actions significantly limits policy performance. Recently, diffusion models have been leveraged for their strong distribution-matching capabilities, enforcing conservatism through behavior policy constraints. However, existing methods often apply indiscriminate regularization to redundant actions in low-quality datasets, resulting in excessive conservatism and an imbalance between the expressiveness and efficiency of diffusion modeling. To address these issues, we propose DIffusion policies with Value-conditional Optimization (DIVO), a novel approach that leverages diffusion models to generate high-quality, broadly covered in-distribution state-action samples while facilitating efficient policy improvement. Specifically, DIVO introduces a binary-weighted mechanism that utilizes the advantage values of actions in the offline dataset to guide diffusion model training. This enables a more precise alignment with the dataset’s distribution while selectively expanding the boundaries of high-advantage actions. During policy improvement, DIVO dynamically filters high-return-potential actions from the diffusion model, effectively guiding the learned policy toward better performance. This approach achieves a critical balance between conservatism and explorability in offline RL. We evaluate DIVO on the D4RL benchmark and compare it against state-of-the-art baselines. Empirical results demonstrate that DIVO achieves superior performance, delivering significant improvements in average returns across locomotion tasks and outperforming existing methods in the challenging AntMaze domain, where sparse rewards pose a major difficulty.

Authors

• Yunchang Ma
• Tenglong Liu
• Yixing Lan
• Xin Yin
• Changxin Zhang
• Xinglong Zhang
• Xin Xu 0001

Keywords

• Training
• Filters
• Reinforcement learning
• Benchmark testing
• Diffusion models
• Reliability
• Optimization
• Intelligent robots
• Overfitting
• Offline Learning
• Offline Reinforcement Learning
• Diffusion Model
• Average Return
• Policy Learning
• Efficient Policies
• Policy Improvement
• Behavior Policy
• Locomotion Tasks
• Value Function
• Optimization Method
• Kullback-Leibler
• Baseline Methods
• Optimal Policy
• Computational Overhead
• Markov Decision Process
• Variational Autoencoder
• Diverse Tasks
• State-action Pair
• Adaptive Optimization
• Maximum Mean Discrepancy
• Conditional Variational Autoencoder
• Final Policy