Dynamic heuristic phased Double Deep Q Network path planning algorithm based on Gaussian mixture regression in discrete traffic environment

In outdoor path planning, effectively balancing the relationship between path capacity and path length remains a significant challenge in current research. To address this challenge, we propose a phased DDQN (Double Deep Q Network) path planning algorithm that integrates Gaussian mixture regression and dynamic heuristics within a discrete traffic environment: GMR-DDQN (Double deep Q network with Gaussian mixture regression). First, the influence of land cover and terrain on vehicle capacity is considered, leading to the formulation of a reward function and the development of a multi-coupled discrete toll cost model. Next, toll costs are incorporated into the DDQN using a greedy strategy, while heuristic methods are applied to generate multiple demonstration trajectories with minimal training. Then, Gaussian mixture regression is employed to gather demonstration trajectory data, construct the proposed path, and generate a buffer-constrained sampling area. The model is further trained to refine the trajectories. Finally, the performance of the algorithm is evaluated in three different traffic environments. Experimental results indicate that, compared to the improved DQN (Deep Q Network) and DDQN algorithms, GMR-DDQN reduces code execution time by 50. 27 %–74. 93 %, shortens path passage time by 27. 12 %–37. 29 %, decreases the average number of steps by 78. 12 %–88. 33 %, and increases the average reward by at least 18, 000 points. Additionally, compared to the unconstrained GMR-DDQN, GMR-DDQN cuts code running time by 25. 79 %–50. 27 %. The results demonstrate that GMR-DDQN can efficiently utilize the road network to enhance traffic efficiency.

Authors

• Ruixin Zhang Tencent Youtu Lab
• Qing Xu
• Kai Sun Xi'an Jiaotong University
• Zhilin He
• Yi Liu City University of Hong Kong
• Youneng Su
• Jingyi Wang
• Xinming Zhu

Keywords

• Deep reinforcement learning
• Imitation learning
• Toll cost modeling
• Heuristic algorithms
• And phased path planning