Language-Guided Object-Centric Diffusion Policy for Generalizable and Collision-Aware Manipulation

Learning from demonstrations faces challenges in generalizing beyond the training data and often lacks collision awareness. This paper introduces Lan-o3dp, a language-guided object-centric diffusion policy framework that can adapt to unseen situations such as cluttered scenes, shifting camera views, and ambiguous similar objects while offering trainingfree collision avoidance and achieving a high success rate with few demonstrations. We train a diffusion model conditioned on 3D point clouds of task-relevant objects to predict the robot's end-effector trajectories, enabling it to complete the tasks. During inference, we incorporate cost optimization into denoising steps to guide the generated trajectory to be collisionfree. We leverage open-set segmentation to obtain the 3D point clouds of related objects. We use a large language model to identify the target objects and possible obstacles by interpreting the user's natural language instructions. To effectively guide the conditional diffusion model using a time-independent cost function, we proposed a novel guided generation mechanism based on the estimated clean trajectories. In the simulation, we showed that diffusion policy based on the object-centric 3D representation achieves a much higher success rate (68. 7%) compared to baselines with simple 2D (39. 3%) and 3D scene (43. 6%) representations across 21 challenging RLBench tasks with only 40 demonstrations. In real-world experiments, we extensively evaluated the generalization in various unseen situations and validated the effectiveness of the proposed zeroshot cost-guided collision avoidance.

Authors

• Hang Li
• Qian Feng
• Zhi Zheng
• Jianxiang Feng
• Zhaopeng Chen
• Alois C. Knoll

Keywords

• Point cloud compression
• Three-dimensional displays
• Robot vision systems
• Training data
• Diffusion models
• Cost function
• End effectors
• Trajectory
• Collision avoidance
• Robots
• Point Cloud
• Diffusion Model
• High Success Rate
• Policy Based
• Target Object
• Language Model
• Similar Objects
• Real-world Experiments
• Camera View
• 3D Point Cloud
• 2D Representation
• Simple 2D
• User Instructions
• Inverse Reinforcement Learning
• Object Point Cloud
• Simulation Experiments
• Similar Shape
• Reversible Process
• Multiple Objects
• Obstacle Avoidance
• Point Cloud Segmentation
• Imitation Learning
• Visual Observation
• Policy Learning
• Gradient Scale
• Front Camera
• Training Policy
• Simple 3D