Spatial Reasoning via Deep Vision Models for Robotic Sequential Manipulation

In this paper, we propose using deep neural architectures (i. e. , vision transformers and ResNet) as heuristics for sequential decision-making in robotic manipulation problems. This formulation enables predicting the subset of objects that are relevant for completing a task. Such problems are often addressed by task and motion planning (TAMP) formulations combining symbolic reasoning and continuous motion planning. In essence, the action-object relationships are resolved for discrete, symbolic decisions that are used to solve manipulation motions (e. g. , via nonlinear trajectory optimization). However, solving long-horizon tasks requires consideration of all possible action-object combinations which limits the scalability of TAMP approaches. To overcome this combinatorial complexity, we introduce a visual perception module integrated with a TAMP-solver. Given a task and an initial image of the scene, the learned model outputs the relevancy of objects to accomplish the task. By incorporating the predictions of the model into a TAMP formulation as a heuristic, the size of the search space is significantly reduced. Results show that our framework finds feasible solutions more efficiently when compared to a state-of-the-art TAMP solver.

Authors

• Hongyou Zhou
• Ingmar Schubert
• Marc Toussaint
• Ozgur S. Oguz

Keywords

• Visualization
• Image color analysis
• Scalability
• Vision sensors
• Transformers
• Cognition
• Planning
• Visual Model
• Robot Manipulator
• Sequential Manipulation
• Search Space
• Path Planning
• Scene Images
• Trajectory Optimization
• Vision Transformer
• Neural Network
• Discretion
• Deep Network
• Artificial Neural Network
• Deep Neural Network
• Deep Learning Models
• Sequence Of Actions
• Adam Optimizer
• Number Of Objects
• Major Objective
• Goal State
• Mixed-integer Programming
• Canonical View
• Feasible Path
• Objects In The Scene
• Relevant Objects
• Residual Neural Network
• Sequence Planning
• Discrete Action
• Tree Search
• Target Object
• Deep Learning