Interactive Multi-Modal Robot Programming

This paper introduces a novel approach to program a robot interactively through a multi-modal interface. The key characteristic of this approach is that the user can provide feedback interactively at any time - during both the programming and the execution phase. The framework takes a three-step approach to the problem: multi-modal recognition, intention interpretation, and prioritized task execution. The multi-modal recognition module translates hand gestures and spontaneous speech into a structured symbolic data stream without abstracting away the user's intent. The intention interpretation module selects the appropriate primitives to generate a task based on the user's input, the system's current state, and robot sensor data. Finally, the prioritized task execution module selects and executes skill primitives based on the system's current state, sensor inputs, and prior tasks. The framework is demonstrated by interactively controlling and programming a vacuum-cleaning robot.

Authors

• Soshi Iba
• Christiaan J. J. Paredis
• Pradeep K. Khosla

Keywords

• Robot programming
• Robot sensing systems
• Feedback
• Speech recognition
• Manipulators
• Robustness
• Computer errors
• Educational robots
• Software systems
• Humans
• Interactive Robots
• Multimodal Robot
• Sensor Data
• Task Execution
• User Input
• Input Modalities
• Usage Intention
• Hand Gestures
• Spontaneous Speech
• Current State Of The System
• Sensor Inputs
• Recognition Module
• Multimodal Interaction
• Navigation
• Vocabulary
• Operation Mode
• Functional Modules
• Hidden Markov Model
• State Machine
• Speech Input
• Path Planning
• Mobile Robot
• Framework Programme
• Hand Velocity
• Robot Control