Task-Driven Graph Attention for Hierarchical Relational Object Navigation

Embodied AI agents in large scenes often need to navigate to find objects. In this work, we study a naturally emerging variant of the object navigation task, hierarchical relational object navigation (HRON), where the goal is to find objects specified by logical predicates organized in a hierarchical structure-objects related to furniture and then to rooms-such as finding an apple on top of a table in the kitchen. Solving such a task requires an efficient representation to reason about object relations and correlate the relations in the environment and in the task goal. HRON in large scenes (e. g. homes) is particularly challenging due to its partial observability and long horizon, which invites solutions that can compactly store the past information while effectively exploring the scene. We demonstrate experimentally that scene graphs are the best-suited representation compared to conventional representations such as images or 2D maps. We propose a solution that uses scene graphs as part of its input and integrates graph neural networks as its backbone, with an integrated task-driven attention mechanism, and demonstrate its better scalability and learning efficiency than state-of-the-art baselines.

Authors

• Michael Lingelbach
• Chengshu Li 0002
• Minjune Hwang
• Andrey Kurenkov
• Alan Lou
• Roberto Martín-Martín
• Ruohan Zhang
• Li Fei-Fei 0001
• Jiajun Wu 0001

Keywords

• Automation
• Navigation
• Scalability
• Graph neural networks
• Task analysis
• Observability
• Artificial intelligence
• Object Relations
• Object Navigation
• Attention Mechanism
• Environment Relationships
• Partial Observation
• Navigation Task
• Kitchen Table
• Scene Graph
• Point Cloud
• Number Of Objects
• Fully-connected Layer
• Target Object
• Visual Search
• Graph Convolutional Network
• Node Features
• Objects In The Scene
• Semantic Map
• Types Of Edges
• Nonexpansive Mapping
• RGB-D Images
• Reinforcement Learning Agent
• Beginning Of Episode
• Privileged Information
• Node Embeddings
• Problem Setup
• Physical Simulation
• Object Instances
• Related Constraints