Task-Oriented Grasp Prediction with Visual-Language Inputs

To perform household tasks, assistive robots receive commands in the form of user language instructions for tool manipulation. The initial stage involves selecting the intended tool (i. e. , object grounding) and grasping it in a task-oriented manner (i. e. , task grounding). Nevertheless, prior researches on visual-language grasping (VLG) focus on object grounding, while disregarding the fine-grained impact of tasks on object grasping. Task-incompatible grasping of a tool will inevitably limit the success of subsequent manipulation steps. Motivated by this problem, this paper proposes GraspCLIP, which addresses the challenge of task grounding in addition to object grounding to enable task-oriented grasp prediction with visual-language inputs. Evaluation on a custom dataset demonstrates that GraspCLIP achieves superior performance over established baselines with object grounding only. The effectiveness of the proposed method is further validated on an assistive robotic arm for grasping previously unseen kitchen tools given the task specification. Our presentation video is available at: https://www.youtube.com/watch?v=e1wfYQPeAXU.

Authors

• Chao Tang 0001
• Dehao Huang
• Lingxiao Meng
• Weiyu Liu
• Hong Zhang 0013

Keywords

• Grounding
• Grasping
• Manipulators
• Assistive robots
• 6-DOF
• Task analysis
• Intelligent robots
• Language Teaching
• Robotic Arm
• Ground Objects
• Robotic Assistance
• Natural Language
• Feature Maps
• Real-world Applications
• Intersection Over Union
• Affordances
• Bounding Box
• Visual Observation
• Learnable Parameters
• Target Object
• Task Instructions
• Target Task
• Object Task
• Word Tokens
• Semantic Gap
• Opening Width
• Sentence Embedding