Learning task outcome prediction for robot control from interactive environments

In order to manage complex tasks such as cooking, future robots need to be action-aware and posses common sense knowledge. For example flipping a pancake requires a robot to know that a spatula has to be under a pancake in order to succeed. We present a novel approach for the extraction and learning of action and common sense knowledge, and developed a game using a robot-simulator with realistic physics for data acquisition. The game environment is a virtual kitchen, in which a user has to create a pancake by pouring pancake-mix on an oven and flipping it using a spatula. The interaction is done by controlling a virtual robot hand with a 3D input sensor. We incorporate a realistic fluid simulation in order to gather appropriate data of the pouring action. Furthermore, we present a task outcome prediction algorithm for this specific system and show how to learn a failure model for the pouring and flipping action.

Authors

• Andrei Haidu
• Daniel Kohlsdorf
• Michael Beetz

Keywords

• Games
• Robot sensing systems
• Hidden Markov models
• Physics
• Ovens
• Liquids
• Results Of Task
• Robot Control
• Simulated Fluid
• Game Environment
• Robotic Hand
• Virtual Hand
• Commonsense Knowledge
• Real Fluid
• Decision Tree
• Surface Tension
• Data Pre-processing
• Second Derivative
• Hidden Markov Model
• Rigid Body
• Repulsive Forces
• Navier Stokes Equations
• Dirac Delta
• Depth Camera
• Task Success
• Fluid Model
• Smoothed Particle Hydrodynamics
• Rigid Body Dynamics
• Physics Engine
• Task Failure
• Game Framework
• Hand Motion