Scene-level Tracking and Reconstruction without Object Priors

We present the first real-time system capable of tracking and reconstructing, individually, every visible object in a given scene, without any form of prior on the rigidness of the objects, texture existence, or object category. In contrast with previous methods such as Co-Fusion and MaskFusion that first segment the scene into individual objects and then process each object independently, the proposed method dynamically segments the non-rigid scene as part of the tracking and reconstruction process. When new measurements indicate topology change, reconstructed models are updated in real-time to reflect that change. Our proposed system can provide the live geometry and deformation of all visible objects in a novel scene in real-time, which makes it possible to be integrated seamlessly into numerous existing robotics applications that rely on object models for grasping and manipulation. The capabilities of the proposed system are demonstrated in challenging scenes that contain multiple rigid and non-rigid objects. Supplementary material, including video, can be found at https://github.com/changhaonan/STAR-no-prior.

Authors

• Haonan Chang
• Abdeslam Boularias

Keywords

• Geometry
• Deformable models
• Visualization
• Stars
• Grasping
• Streaming media
• Real-time systems
• Uniform Prior
• Topological Changes
• Individual Objects
• Robotic Applications
• Appended
• K-nearest Neighbor
• Point Cloud
• Nodes In The Graph
• Pre-trained Network
• Depth Measurements
• Model Geometry
• Recent Techniques
• Index Function
• Object Tracking
• Objects In The Scene
• Initialization Strategy
• Camera Pose
• Dynamic Scenes
• Scene Reconstruction
• RGB-D Images
• Warp Field
• Real-time Reconstruction
• Global Geometry
• Simultaneous Tracking
• Time Step
• Internal Constraints
• Tracking Problem
• Measurement Noise
• Robot Manipulator
• Simultaneous Reconstruction