Synthesizing Robot Manipulation Programs from a Single Observed Human Demonstration

Programming by Demonstration (PbD) lets users with little technical background program a wide variety of manipulation tasks for robots, but it should be as intuitive as possible for users while requiring as little time as possible. In this paper, we present a Programming by Demonstration system that synthesizes manipulation programs from a single observed demonstration, allowing users to program new tasks for a robot simply by performing the task once themselves. A human-in-the-loop interface helps users make corrections to the perceptual state as needed. We introduce Object Interaction Programs as a representation of multi-object, bimanual manipulation tasks and present algorithms for extracting programs from observed demonstrations and transferring programs to a robot to perform the task in a new scene. We demonstrate the expressivity and generalizability of our approach through an evaluation on a benchmark of complex tasks.

Authors

• Justin Huang
• Dieter Fox
• Maya Cakmak

Keywords

• Hands
• Programming
• Benchmark testing
• Robot sensing systems
• Sensor systems
• Human in the loop
• Sensors
• Object tracking
• Robots
• Intelligent robots
• Manipulation Tasks
• Object Interaction
• Intersection Over Union
• Second Category
• Use Of Markers
• Motion Capture
• Small Objects
• Path Planning
• Trials Of Task
• Depth Camera
• Use Of Complexes
• Coordinate Frame
• Objects In The Scene
• Program Execution
• Inverse Kinematics
• Transfer Task
• Object Pose
• Tennis Ball
• Object Trajectory
• Target Pose
• End-effector Pose
• Task Representations
• Work In This Area
• Radially Symmetric
• Video Presentation
• End Of Segment