General-Purpose Clothes Manipulation with Semantic Keypoints

Clothes manipulation is a critical capability for household robots; yet, existing methods are often confined to specific tasks, such as folding or flattening, due to the complex high-dimensional geometry of deformable fabric. This paper presents CLothes mAnipulation with Semantic keyPoints (CLASP) for general-purpose clothes manipulation, which enables the robot to perform diverse manipulation tasks over different types of clothes. The key idea of CLASP is semantic keypoints-e. g. , “right shoulder”, “left sleeve”, etc. -a sparse spatial-semantic representation that is salient for both perception and action. Semantic keypoints of clothes can be effectively extracted from depth images and are sufficient to represent a broad range of clothes manipulation policies. CLASP leverages semantic keypoints to bridge LLM-powered task planning and low-level action execution in a two-level hierarchy. Extensive simulation experiments show that CLASP outperforms baseline methods across diverse clothes types in both seen and unseen tasks. Further, experiments with a Kinova dual-arm system on four distinct tasks-folding, flattening, hanging, and placing-confirm CLASP's performance on a real robot.

Authors

• Yuhong Deng
• David Hsu

Keywords

• Geometry
• Deep learning
• Deformation
• Semantics
• Pipelines
• Fabrics
• Planning
• Spatiotemporal phenomena
• Robots
• Commonsense reasoning
• Semantic Keypoints
• Specific Tasks
• Simulation Experiments
• Baseline Methods
• Depth Images
• Manipulation Tasks
• Action Execution
• Task Planning
• Contact Point
• Language Teaching
• Average Precision
• Encoder-decoder
• Range Of Tasks
• Hierarchical Method
• Multi-task Learning
• Mean Average Precision
• General Representation
• Masked Images
• Description Language
• Deformable Objects
• Reconstruction Stage
• Commonsense Knowledge
• Keypoint Detection
• General-purpose Method
• Spatiotemporal Representation
• Partial Observation
• Wide Range Of Tasks
• Task Requirements
• Data-driven Methods