DNAct: Diffusion Guided Multi-Task 3D Policy Learning

This paper presents DNAct, a language-conditioned multi-task policy framework that integrates neural rendering pre-training and diffusion training to enforce multi-modality learning in action sequence spaces. To learn a generalizable multi-task policy with few demonstrations, the pre-training phase of DNAct leverages neural rendering to distill 2D semantic features from foundation models such as Stable Diffusion to a 3D space, which provides a comprehensive semantic understanding regarding the scene. Consequently, it allows various applications for challenging robotic tasks requiring rich 3D semantics and accurate geometry. Furthermore, we introduce a novel approach utilizing diffusion training to learn a vision and language feature that encapsulates the inherent multi-modality in the multi-task demonstrations. By reconstructing the action sequences from different tasks via the diffusion process, the model is capable of distinguishing different modalities and thus improving the robustness and the generalizability of the learned representation. DNAct significantly surpasses SOTA NeRF-based multi-task manipulation approaches with over 30% improvement in success rate. Videos are available on dnact.github.io

Authors

• Ge Yan
• Yueh-Hua Wu
• Xiaolong Wang

Keywords

• Training
• Three-dimensional displays
• Semantics
• Diffusion processes
• Multitasking
• Rendering (computer graphics)
• Neural radiance field
• Robustness
• Intelligent robots
• Videos
• Multi-task Learning
• Policy Learning
• Diffusion Process
• Sequence Of Actions
• 3D Space
• Representation Learning
• Semantic Features
• Robotic Tasks
• Foundation Model
• Semantic Understanding
• Improve Success Rates
• Accurate Geometry
• Denoising
• Semantic Information
• Diffusion Model
• Feature Points
• Activity Prediction
• 3D Point
• 3D Representation
• Robot Manipulator
• Multi-view Images
• Neural Field
• View Synthesis
• Robot Experiments
• Real Robot
• Policy Network
• Robot Learning
• View Direction
• Active Inference
• Point Cloud Features