Mixture-attention Siamese transformer for video polyp segmentation

Accurate segmentation of polyps from colonoscopy videos is of great significance to polyp treatment and early prevention of colorectal cancer. However, it is challenging due to the difficulties associated with modeling long-range spatio-temporal relationships within a colonoscopy video. In this paper, we address this challenging task with a novel Mixture-Attention Siamese Transformer (MAST), which explicitly models the long-range spatio-temporal relationships with a mixture-attention mechanism for accurate polyp segmentation. Specifically, we first construct a Siamese transformer architecture to jointly encode paired video frames for their feature representations. We then design a mixture-attention module to exploit the intra-frame and inter-frame correlations, enhancing the features with rich spatio-temporal relationships. Finally, the enhanced features are fed to two parallel decoders for predicting the segmentation maps. Extensive experiments on the large-scale SUN-SEG benchmark demonstrate the superior performance of MAST in comparison with the cutting-edge competitors. Our code is publicly available at https: //github. com/Junqing-Yang/MAST.

Authors

• Geng Chen College of CSSE, Guangdong Key Laboratory of Intelligent Information Processing, Shenzhen University State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications
• Junqing Yang
• Xiaozhou Pu
• Ge-Peng Ji
• Huan Xiong Jilin University Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)
• Yongsheng Pan National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University, Xi'an, China
• Hengfei Cui
• Yong Xia National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology, School of Computer Science and Engineering, Northwestern Polytechnical University, Xi'an, China Research & Development Institute of Northwestern Polytechnical University in Shenzhen, Shenzhen, China Ningbo Institute of Northwestern Polytechnical University, Ningbo, China

Keywords

• Video polyp segmentation
• Colonoscopy
• Attention mechanism
• Transformer