Learning Generalizable Feature Fields for Mobile Manipulation

An open problem in mobile manipulation is how to represent objects and scenes in a unified manner so that robots can use both for navigation and manipulation. The latter requires capturing intricate geometry while understanding fine-grained semantics, whereas the former involves capturing the complexity inherent at an expansive physical scale. In this work, we present GeFF (Generalizable Feature Fields), a scene-level generalizable neural feature field that acts as a unified representation for both navigation and manipulation that performs in real-time. To do so, we treat generative novel view synthesis as a pre-training task, and then align the resulting rich scene priors with natural language via CLIP feature distillation. We demonstrate the effectiveness of this approach by deploying GeFF on a quadrupedal robot equipped with a manipulator. We quantitatively evaluate GeFF’s ability for open-vocabulary object-/part-level manipulation and show that GeFF outperforms point-based baselines in runtime and storage-accuracy trade-offs, with qualitative examples of semantics-aware navigation and articulated object manipulation.

Authors

• Ri-Zhao Qiu
• Yafei Hu
• Yuchen Song
• Ge Yang
• Yang Fu
• Jianglong Ye
• Jiteng Mu
• Ruihan Yang
• Nikolay Atanasov 0001
• Sebastian A. Scherer
• Xiaolong Wang 0004

Keywords

• Geometry
• Runtime
• Navigation
• Semantics
• Natural languages
• Manipulators
• Real-time systems
• Complexity theory
• Quadrupedal robots
• Intelligent robots
• Mobile Manipulator
• Unified Representation
• Quadrupedal
• Neural Field
• View Synthesis
• Quadruped Robot
• Grid Cells
• Point Cloud
• Path Planning
• Semantic Features
• Mobile Robot
• Latent Representation
• Query Language
• Implicit Method
• 2D Feature
• Scene Changes
• Scene Representation
• Signed Distance Function
• Text Query
• Occupancy Map