AAMAS Conference 2026 Conference Paper
Learning from Delay Distributions: A New Representation for Delay-Aware Reinforcement Learning
- Zhuoru Yu
- Chenchen Fu
- Hengkai Zhong
- Wanyuan Wang
- Weiwei Wu
- Chun Jason Xue
Delay remains a significant challenge for applying deep reinforcement learning (DRL) in real-world scenarios. Existing delay-aware DRL methods primarily rely on state augmentation to restore the Markov property in delayed environments, yet often assume the prior knowledge of the exact delay values and suffer from performance degradation in random delay environments. However, we observe that the random delays in real-world often follow specific statistical patterns. Based on this observation, we propose a novel method that leverages distributions to represent value functions, enabling a more accurate modeling of delay uncertainty beyond traditional expectation-based methods. Building upon delay distribution properties, we introduce a stochastic delay representation mechanism to reconstruct precise returns in delayed environments and prove its convergence to the optimal policy. Finally, we apply these techniques to design the delay-aware distributional actor-critic (D2AC) DRL framework. Experimental results show that D2AC significantly outperforms state-of-the-art delay-aware DRL methods across various random delay distributions in Mu- JoCo continuous control tasks. Open source code and appendix are available at: https: //github. com/COOLAS-CS/D2AC.