Where It Moves, It Matters: Referring Surgical Instrument Segmentation via Motion

Enabling intuitive, language-driven interaction with surgical scenes is a critical step toward intelligent operating rooms and autonomous surgical robotic assistance. However, the task of referring segmentation, localizing surgical instruments based on natural language descriptions, remains underexplored in surgical videos, with existing approaches struggling to generalize due to reliance on static visual cues and predefined instrument names. In this work, we introduce SurgRef, a novel motion-guided framework that grounds free-form language expressions in instrument motion, capturing how tools move and interact across time, rather than what they look like. This allows models to understand and segment instruments even under occlusion, ambiguity, or unfamiliar terminology. To train and evaluate SurgRef, we present Ref-IMotion, a diverse, multi-institutional video dataset with dense spatiotemporal masks and rich motion-centric expressions. SurgRef achieves state-of-the-art accuracy and generalization across surgical procedures, setting a new benchmark for robust, language-driven surgical video segmentation.

Authors

• Meng Wei
• Kun Yuan Technical University of Munich, Germany Munich Center for Machine Learning, Germany University of Strasbourg, France
• Shi Li
• Yue Zhou Shanghai University
• Long Bai School of Computer Science and Technology, University of Chinese Academy of Sciences
• Nassir Navab
• Hongliang Ren The Chinese University of Hong Kong The Chinese University of Hong Kong, Shenzhen Research Institute
• Hong Joo Lee Technische Universität München
• Tom Vercauteren King's College London
• Nicolas Padoy University of Strasbourg, France IHU Strasbourg, France

Keywords

No keywords are indexed for this paper.