Sequential Multi-Object Grasping with One Dexterous Hand

Sequentially grasping multiple objects with multi-fingered hands is common in daily life, where humans can fully leverage the dexterity of their hands to enclose multiple objects. However, the diversity of object geometries and the complex contact interactions required for high-DOF hands to grasp one object while enclosing another make sequential multi-object grasping challenging for robots. In this paper, we propose SeqMultiGrasp, a system for sequentially grasping objects with a four-fingered Allegro Hand. We focus on sequentially grasping two objects, ensuring that the hand fully encloses one object before lifting it and then grasps the second object without dropping the first. Our system first synthesizes single-object grasp candidates, where each grasp is constrained to use only a subset of the hand’s links. These grasps are then validated in a physics simulator to ensure stability and feasibility. Next, we merge the validated single-object grasp poses to construct multi-object grasp configurations. For real-world deployment, we train a diffusion model conditioned on point clouds to propose grasp poses, followed by a heuristic-based execution strategy. We test our system using 8 × 8 object combinations in simulation and 6 × 3 object combinations in real. Our diffusion-based grasp model obtains an average success rate of 65. 8% over 1, 600 simulation trials and 56. 7% over 90 real-world trials, suggesting that it is a promising approach for sequential multi-object grasping with multi-fingered hands. Supplementary material is available on our project website: https://hesic73.github.io/SeqMultiGrasp.

Authors

• Sicheng He
• Zeyu Shangguan
• Kuanning Wang
• Yongchong Gu
• Yuqian Fu
• Yanwei Fu 0001
• Daniel Seita

Keywords

• Hands
• Point cloud compression
• Geometry
• Shape
• Grasping
• Diffusion models
• Stability analysis
• Data models
• Physics
• Intelligent robots
• Dexterous Hand
• Point Cloud
• Diffusion Model
• Multiple Objects
• Physical Simulation
• Average Success Rate
• Simulation Experiments
• Contact Point
• Energy Function
• Singular Value Decomposition
• Failure Modes
• Path Planning
• Joint Position
• Real-world Experiments
• End-effector
• Object Pairs
• Robotic Hand
• Dexterous Manipulation
• Object Point Cloud