Closed Loop Interactive Embodied Reasoning for Robot Manipulation

Embodied reasoning systems integrate robotic hardware and cognitive processes to perform complex tasks, typically in response to a natural language query about a specific physical environment. This usually involves changing the belief about the scene or physically interacting and changing the scene (e. g. sort the objects from lightest to heaviest). In order to facilitate the development of such systems we introduce a new modular Closed Loop Interactive Embodied Reasoning (CLIER) approach that takes into account the measurements of non-visual object properties, changes in the scene caused by external disturbances as well as uncertain outcomes of robotic actions. CLIER performs multi-modal reasoning and action planning and generates a sequence of primitive actions that can be executed by a robot manipulator. Our method operates in a closed loop, responding to changes in the environment. Our approach is developed with the use of MuBle simulation environment and tested in $\mathbf{1 0}$ interactive benchmark scenarios. We extensively evaluate our reasoning approach in simulation and in real-world manipulation tasks with a success rate above $\mathbf{7 6 \%}$ and 64%, respectively.

Authors

• Michal Nazarczuk
• Jan Kristof Behrens
• Karla Stépánová
• Matej Hoffmann
• Krystian Mikolajczyk

Keywords

• Visualization
• Cognitive processes
• Natural languages
• Measurement uncertainty
• Manipulators
• Cognition
• Hardware
• Planning
• Robots
• Physics
• Robot Manipulator
• Action Plan
• Natural Language
• Sequence Of Actions
• Simulation Environment
• Object Properties
• Manipulation Tasks
• Reasonable Approach
• Real-world Tasks
• Scene Changes
• Development Of Such Systems
• Stiffness
• Physical Measures
• Control Signal
• Task Completion
• Types Of Errors
• Visual Observation
• Pose Estimation
• End-effector
• Objects In The Scene
• Scene Graph
• Real Scenes
• Objective Weight
• End-effector Pose
• Robotic Tasks
• Visual Reasoning
• Part Of Table
• Real Robot
• Joint Limits
• Loop Detection