Diffusion-PbD: Generalizable Robot Programming by Demonstration with Diffusion Features

Programming by Demonstration (PbD) is an intuitive technique for programming robot manipulation skills by demonstrating the desired behavior. However, most existing approaches either require extensive demonstrations or fail to generalize beyond their initial demonstration conditions. We introduce Diffusion-PbD, a novel approach to PbD that enables users to synthesize generalizable robot manipulation skills from a single demonstration by utilizing the representations captured by pre-trained visual foundation models. At demonstration time, hand and object detection priors are used to extract waypoints from the human demonstrations anchored to reference points in the scene. At execution time, features from pre-trained diffusion models are leveraged to identify corresponding reference points in new observations. We validate this approach through a series of real-world robot experiments, showing that Diffusion-PbD is applicable to a wide range of manipulation tasks and has strong ability to generalize to unseen objects, camera viewpoints, and scenes. Code and supplementary videos can be found at https://diffusion-pbd.github.io

Authors

• Michael Murray
• Entong Su
• Maya Cakmak

Keywords

• Visualization
• Foundation models
• Dynamics
• Robot vision systems
• Feature extraction
• Robustness
• Trajectory
• Robots
• Videos
• Robot programming
• Diffusion Characteristics
• Robot Programming By Demonstration
• Diffusion Model
• Range Of Tasks
• Manipulation Tasks
• Robot Manipulator
• Wide Range Of Tasks
• Foundation Model
• Scene Point
• Unseen Objects
• Use Of Tools
• Index Finger
• Bounding Box
• Target Object
• Depth Map
• 3D Point
• Depth Camera
• Objective Structured
• Image Synthesis
• Scene Structure
• Objects In The Scene
• Coffee Machine
• RGB-D Images
• Passive Observation
• Executive Skills
• Task Structure
• Object Bounding Boxes
• Deformable Objects
• Motion Generation