Prospection: Interpretable plans from language by predicting the future

High-level human instructions often correspond to behaviors with multiple implicit steps. In order for robots to be useful in the real world, they must be able to to reason over both motions and intermediate goals implied by human instructions. In this work, we propose a framework for learning representations that convert from a natural-language command to a sequence of intermediate goals for execution on a robot. A key feature of this framework is prospection, training an agent not just to correctly execute the prescribed command, but to predict a horizon of consequences of an action before taking it. We demonstrate the fidelity of plans generated by our framework when interpreting real, crowd-sourced natural language commands for a robot in simulated scenes.

Authors

• Chris Paxton 0001
• Yonatan Bisk
• Jesse Thomason
• Arunkumar Byravan
• Dieter Fox

Keywords

• Task analysis
• Robots
• Training
• Natural languages
• Visualization
• Planning
• Predictive models
• Prospection
• Natural Language
• Representation Learning
• Command Of Language
• Time Step
• Sequence Of Actions
• Global Status
• Latent Space
• Hidden State
• Mechanical Turk
• High Level Of Control
• Latent State
• End-effector
• Prediction Module
• Task Planning
• Low-level Control
• L2 Loss
• Action Verbs
• Multiple Time Steps
• Cartesian Position
• Yellow Block
• Robotic Agents
• End-effector Pose
• Language Learning
• Semantic
• Planning Horizon