A Simple Image Segmentation Framework via In-Context Examples

Recently, there have been explorations of generalist segmentation models that can effectively tackle a variety of image segmentation tasks within a unified in-context learning framework. However, these methods still struggle with task ambiguity in in-context segmentation, as not all in-context examples can accurately convey the task information. In order to address this issue, we present SINE, a simple image $\textbf{S}$egmentation framework utilizing $\textbf{in}$-context $\textbf{e}$xamples. Our approach leverages a Transformer encoder-decoder structure, where the encoder provides high-quality image representations, and the decoder is designed to yield multiple task-specific output masks to eliminate task ambiguity effectively. Specifically, we introduce an In-context Interaction module to complement in-context information and produce correlations between the target image and the in-context example and a Matching Transformer that uses fixed matching and a Hungarian algorithm to eliminate differences between different tasks. In addition, we have further perfected the current evaluation system for in-context image segmentation, aiming to facilitate a holistic appraisal of these models. Experiments on various segmentation tasks show the effectiveness of the proposed method.

Authors

• Yang Liu College of Computer Science and Software Engineering, Shenzhen University
• Chenchen Jing Zhejiang University
• Hengtao Li
• Muzhi Zhu Zhejiang University
• Hao Chen The Key Laboratory of Cognition and Decision Intelligence for Complex Systems, Institute of Automation, CASIA & School of Artificial Intelligence, UCAS
• Xinlong Wang
• Chunhua Shen College of Computer Science and Technology, Zhejiang University

Keywords

No keywords are indexed for this paper.