MesaTask: Towards Task-Driven Tabletop Scene Generation via 3D Spatial Reasoning

The ability of robots to interpret human instructions and execute manipulation tasks necessitates the availability of task-relevant tabletop scenes for training. However, traditional methods for creating these scenes rely on time-consuming manual layout design or purely randomized layouts, which are limited in terms of plausibility or alignment with the tasks. In this paper, we formulate a novel task, namely task-oriented tabletop scene generation, which poses significant challenges due to the substantial gap between high-level task instructions and the tabletop scenes. To support research on such a challenging task, we introduce \textbf{MesaTask-10K}, a large-scale dataset comprising approximately 10, 700 synthetic tabletop scenes with \emph{manually crafted layouts} that ensure realistic layouts and intricate inter-object relations. To bridge the gap between tasks and scenes, we propose a \textbf{Spatial Reasoning Chain} that decomposes the generation process into object inference, spatial interrelation reasoning, and scene graph construction for the final 3D layout. We present \textbf{MesaTask}, an LLM-based framework that utilizes this reasoning chain and is further enhanced with DPO algorithms to generate physically plausible tabletop scenes that align well with given task descriptions. Exhaustive experiments demonstrate the superior performance of MesaTask compared to baselines in generating task-conforming tabletop scenes with realistic layouts.

Authors

• Jinkun Hao Shanghai Jiao Tong University
• Naifu Liang
• Zhen Luo
• Xudong XU
• Weipeng Zhong
• Ran Yi Shanghai Jiao Tong University
• Yichen Jin
• Zhaoyang Lyu
• Feng Zheng Southern University of Science and Technology
• Lizhuang Ma Shanghai Jiao Tong University, Shanghai, China East China Normal University, Shanghai, China MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China
• Jiangmiao Pang

Keywords

No keywords are indexed for this paper.