Relationship Oriented Semantic Scene Understanding for Daily Manipulation Tasks

Assistive robot systems have been developed to help people accomplish daily manipulation tasks especially for those with disabilities, where scene understanding plays a crucial role in enabling robots to interpret the surroundings and behave accordingly. Most of the current systems approach scene understanding without considering the functional dependencies between objects. However, it is only valuable to interact with some objects when their function-relevant counterparts are considered. In this paper, we augment an assistive robotic arm system with an end-to-end semantic relationship reasoning model. It incorporates functional relationships between pairs of objects for semantic scene understanding. To ensure good generalization to unseen objects and relationships, the model works in a category-agnostic manner. We evaluate our design and three baseline methods on a self-collected benchmark with two levels of difficulty. To further demonstrate the effectiveness, the model is integrated with a symbolic planner for goal-oriented, multi-step manipulation task on a real-world assistive robotic arm platform.

Authors

• Chao Tang 0001
• Jingwen Yu
• Weinan Chen
• Bingyi Xia
• Hong Zhang 0013

Keywords

• Visualization
• Semantics
• Pipelines
• Benchmark testing
• Manipulators
• Assistive robots
• Cognition
• Manipulation Tasks
• Daily Tasks
• Scene Understanding
• Semantic Scene Understanding
• Daily Manipulation
• Functional Independence
• Difficulty Level
• Robotic System
• Robotic Arm
• Assistance Systems
• Robotic Platform
• Object Pairs
• Robotic Assistance
• Semantic Understanding
• Unseen Objects
• Training Set
• Computer Vision
• Amyotrophic Lateral Sclerosis
• Object Detection
• Bounding Box
• Original Feature Map
• Object Features
• Scene Graph
• Planning Module
• Category Labels
• Perception Module
• Objects In The Scene
• Robot Manipulator
• Object Proposals
• Absence Of Labels