Interactive Robot Knowledge Patching Using Augmented Reality

We present a novel Augmented Reality (AR) approach, through Microsoft HoloLens, to address the challenging problems of diagnosing, teaching, and patching interpretable knowledge of a robot. A Temporal And-Or graph (T-AOG) of opening bottles is learned from human demonstration and programmed to the robot. This representation yields a hierarchical structure that captures the compositional nature of the given task, which is highly interpretable for the users. By visualizing the knowledge structure represented by a T-AOG and the decision making process by parsing the T-AOG, the user can intuitively understand what the robot knows, supervise the robot's action planner, and monitor visually latent robot states ( e. g. , the force exerted during interactions). Given a new task, through such comprehensive visualizations of robot's inner functioning, users can quickly identify the reasons of failures, interactively teach the robot with a new action, and patch it to the current knowledge structure. In this way, the robot is capable of solving similar but new tasks only through minor modifications provided by the users interactively. This process demonstrates the interpretability of our knowledge representation and the effectiveness of the AR interface.

Authors

• Hangxin Liu
• Yaofang Zhang
• Wenwen Si
• Xu Xie 0001
• Yixin Zhu 0001
• Song-Chun Zhu

Keywords

• Task analysis
• Decision making
• Knowledge representation
• Visualization
• Robot sensing systems
• Grammar
• Knowledge Of The Robot
• Decision-making Process
• Parsing
• Knowledge Of Structure
• Temporal Graph
• Deep Neural Network
• Sequence Of Actions
• Additional Activities
• Sensory Information
• System Architecture
• Markov Decision Process
• Child Nodes
• Terminal Nodes
• Robotic Platform
• Forced Response
• Head-mounted Display
• Gesture Recognition
• Imitation Learning
• Robot Operating System
• Inverse Reinforcement Learning
• Parse Tree
• Interpretable Representation
• Pill Bottle
• Augmented Reality System
• Rich History
• Types Of Outcomes
• Nature Of Interactions