GRaD-Nav: Efficiently Learning Visual Drone Navigation with Gaussian Radiance Fields and Differentiable Dynamics

Autonomous visual navigation is an essential element in robot autonomy. Reinforcement learning (RL) offers a promising policy training paradigm. However, existing RL methods suffer from high sample complexity, poor sim-to-real transfer, and limited runtime adaptability. These problems are particularly challenging for drones, with complex nonlinear and unstable dynamics, and strong dynamic coupling between control and perception. In this paper, we propose a novel framework that integrates 3D Gaussian Splatting (3DGS) with differentiable deep reinforcement learning (DDRL) to train vision-based drone navigation policies. By leveraging high-fidelity 3D scene representations and differentiable simulation, our method improves sample efficiency and sim-to-real transfer. Additionally, we incorporate a Context-aided Estimator Network (CENet) to adapt to environmental variations at runtime. Moreover, by curriculum training in a mixture of different surrounding environments, we achieve in-task generalization, the ability to solve new instances of a task not seen during training. Drone hardware experiments demonstrate our method’s high training efficiency compared to state-of-the-art RL methods, zero shot sim-to-real transfer for real robot deployment without fine tuning, and ability to adapt to new instances within the same task class (e. g. to fly through a gate at different locations with different distractors in the environment). Our simulator and training framework are open-sourced at: https://github.com/Qianzhong-Chen/grad_nav.

Authors

• Qianzhong Chen
• Jiankai Sun
• Naixiang Gao
• JunEn Low
• Timothy Chen
• Mac Schwager

Keywords

• Training
• Visualization
• Three-dimensional displays
• Runtime
• Navigation
• Logic gates
• Robustness
• Trajectory
• Tuning
• Drones
• Differences In Dynamics
• Drone Navigation
• Sampling Efficiency
• Training Efficiency
• Deep Reinforcement Learning
• Training Curriculum
• Machine Vision
• Real Robot
• Training Policy
• Environmental Distractions
• Learning Algorithms
• Convolutional Neural Network
• Angular Velocity
• Point Cloud
• Diverse Environments
• Multilayer Perceptron
• Depth Images
• Reward Function
• Linear Velocity
• Domain Adaptation
• Backpropagation Through Time
• Reinforcement Learning Algorithm
• Policy Network
• Proximal Policy Optimization
• Efficient Navigation
• Reference Trajectory
• Safe Navigation
• Obstacle Avoidance
• Short Horizon
• SqueezeNet