Moving SLAM: Fully Unsupervised Deep Learning in Non-Rigid Scenes

We propose a new deep learning framework to decompose monocular videos into 3D geometry (camera pose and depth), moving objects, and their motions, with no supervision. We build upon the idea of view synthesis, which uses classical camera geometry to re-render a source image from a different point-of-view to obtain supervisory signals, specified by a predicted relative 6-degree-of-freedom pose and depth map. However, the typical view synthesis equations rely on a strong assumption: that objects in scenes do not move. This rigid-world assumption limits the predictive power, and rules out learning about objects automatically. We propose a simple solution: minimize the synthesis error on small local regions of the image instead. While the scene as a whole may be non-rigid, it is always possible to find small regions that are approximately rigid, such as inside a moving object. Our network can learn a dense pose map describing poses for each local region. This represents a significantly richer model, including 6D object motions, with little additional complexity. We establish very competitive results on unsupervised odometry and depth prediction on KITTI. We also demonstrate new capabilities on EPIC-Kitchens, a challenging dataset of indoor videos, where there is no ground truth information for depth, odometry, object segmentation or motion - yet all are recovered automatically by our approach.

Authors

• Dan Xu 0002
• Andrea Vedaldi
• João F. Henriques

Keywords

• Geometry
• Deep learning
• Three-dimensional displays
• Simultaneous localization and mapping
• Motion segmentation
• Object segmentation
• Cameras
• Non-rigid Scene
• Depth Map
• Object Motion
• 3D Geometry
• Objects In The Scene
• Camera Pose
• Depth Prediction
• View Synthesis
• Supervisory Signal
• Neural Network
• Qualitative Results
• Unsupervised Learning
• Estimation Performance
• Large-scale Datasets
• Inertial Measurement Unit
• Pose Estimation
• Depth Estimation
• Relative Pose
• Visual Odometry
• Monocular Depth Estimation
• Self-supervised Learning
• CNN Backbone
• Target Frame
• Static Background
• Motion Estimation
• Constant Area
• Rigid Model