AutoNeRF: Training Implicit Scene Representations with Autonomous Agents

Implicit representations such as Neural Radiance Fields (NeRF) allow to map color, density and semantics in a 3D scene through a continuous neural function. However, these models typically require manual and careful human data collection for training. This paper addresses the problem of active exploration for autonomous NeRF construction. We study how an agent can learn to efficiently explore an unknown 3D environment so that the data collected during autonomous exploration enables the learning of a high-quality neural implicit map representation. The quality of the learned representation is evaluated on four robotics-related downstream tasks: classical viewpoint rendering, map reconstruction, planning, and pose refinement. We compare the impact of different exploration strategies including frontier-based and learning-based approaches (end-to-end and modular) with different reward functions tailored to this problem. Empirical results show that NeRFs can be trained on actively collected data using just a single episode of experience in an unseen environment and that AutoNeRF, a modular exploration policy trained with reinforcement learning, enables obtaining a higher-quality NeRF for the considered downstream robotic tasks. Finally, we show that with AutoNeRF an agent can be deployed to a previously unknown scene and then automatically improve its navigation performance by adapting to the scene through a cycle of exploration, reconstruction, and policy finetuning.

Authors

• Pierre Marza
• Laëtitia Matignon
• Olivier Simonin 0001
• Dhruv Batra
• Christian Wolf 0001
• Devendra Singh Chaplot

Keywords

• Training
• Three-dimensional displays
• Navigation
• Semantics
• Robot vision systems
• Training data
• Reinforcement learning
• Neural radiance field
• Rendering (computer graphics)
• Planning
• Autonomous Agents
• Scene Representation
• Implicit Scene Representation
• Neural Representations
• Learning-based Approaches
• Reward Function
• 3D Environment
• Quality Of Representations
• 3D Scene
• Training Policy
• Implicit Representation
• Unseen Environments
• Fine-tuned
• Semantic Information
• Path Planning
• Start Position
• Global Policy
• 3D Position
• Pose Estimation
• Semantic Map
• Neural Field
• Camera Pose
• Mask R-CNN
• Deep Reinforcement Learning
• Nonexpansive Mapping
• View Synthesis
• Training Scenes
• Odometry
• Scene Reconstruction