AAMAS Conference 2023 Conference Paper
Modeling Dynamic Environments with Scene Graph Memory
- Andrey Kurenkov
- Michael Lingelbach
- Tanmay Agarwal
- Chengshu Li
- Emily Jin
- Ruohan Zhang
- Li Fei-Fei
- Jiajun Wu
Embodied AI agents operating in dynamic environments often need to predict object locations to make informed decisions. We propose a method for doing this via link prediction on partially observable dynamic graphs. We represent the agent’s accumulated set of observations in a data structure called a Scene Graph Memory (SGM), combine this data structure with a neural net architecture we call Node Edge Predictor (NEP), and show that it can be trained to predict the locations of objects in a variety of environments with diverse object movement dynamics. To evaluate our method, we implement the Dynamic Household Simulator, a novel benchmark which enables sampling of diverse dynamic scene graphs that follow the semantic patterns typically seen at peoples’ homes. We demonstrate that our method outperforms baselines both in terms of quickly adapting to the dynamics of a new scene and in terms of its overall accuracy.