Modeling Dynamic Environments with Scene Graph Memory

Embodied AI agents operating in dynamic environments often need to predict object locations to make informed decisions. We propose a method for doing this via link prediction on partially observable dynamic graphs. We represent the agent’s accumulated set of observations in a data structure called a Scene Graph Memory (SGM), combine this data structure with a neural net architecture we call Node Edge Predictor (NEP), and show that it can be trained to predict the locations of objects in a variety of environments with diverse object movement dynamics. To evaluate our method, we implement the Dynamic Household Simulator, a novel benchmark which enables sampling of diverse dynamic scene graphs that follow the semantic patterns typically seen at peoples’ homes. We demonstrate that our method outperforms baselines both in terms of quickly adapting to the dynamics of a new scene and in terms of its overall accuracy.

Authors

• Andrey Kurenkov Stanford University
• Michael Lingelbach Stanford University
• Tanmay Agarwal Stanford University
• Chengshu Li Stanford University
• Emily Jin
• Ruohan Zhang
• Li Fei-Fei
• Jiajun Wu University of Calgary
• Silvio Savarese Salesforce AI Research
• Roberto Martín-Martín The University of Texas at Austin

Keywords

• Robotics
• Graph Neural Network
• Embodied AI
• Scene Graph
• Link
• Prediction
• State Representation