Versatile Vision-Language Model for 3D Computed Tomography

Representation learning serves as a foundational component of medical vision-language models (MVLMs), enabling cross-modal alignment, semantic consistency, and enhanced generalization capabilities for downstream tasks. As generalist models rapidly evolve, there is a pressing need to unify diverse downstream tasks, such as diagnosis, segmentation, report generation, and multiple choice within a cohesive framework, demanding more efficient and versatile visual representation learning. However, current MVLMs predominately follow CLIP-style vision pretraining, failing to leverage heterogeneous data resources with multi-dimensional imaging and diverse annotation forms. And there lacks systematic analysis of efficient vision encoder design across varied downstream applications, including diagnosis, segmentation, and text generation tasks, particularly for volumetric imaging like Computed Tomography (CT). Besides, current MVLMs exhibit constrained voxel-level capabilities, lacking effective multi-task instruction tuning framework capable of achieving robust performance across various downstream tasks. To address these challenges, we propose CTInstruct, a novel MVLM employing a hybrid ResNet-ViT encoder with multi-granular vision-language pretraining for efficient heterogeneous data modeling, and unified instruction tuning that jointly optimizes discriminative, generative, and voxel-level reasoning for volumetric medical imaging. CTInstruct achieves SOTA performance across 8 CT benchmarks, setting a new standard for data-efficient multimodal learning in medical imaging.

Authors

• Jiayu Lei University of Science and Technology of China, Anhui, China Shanghai Artificial Intelligence Laboratory, Shanghai, China
• Ziqing Fan School of Artificial Intelligence, Shanghai Jiao Tong University, Shanghai, China Shanghai Artificial Intelligence Laboratory, Shanghai, China
• Yanyong Zhang School of Computer Science and Technology, University of Science and Technology of China, Hefei, China Institute of Artificial Intelligence, Hefei Comprehensive National Science Center, Hefei, China
• Weidi Xie School of Artificial Intelligence, Shanghai Jiao Tong University, China
• Ya Zhang Shanghai Jiao Tong University Shanghai Artificial Intelligence Laboratory
• Yanfeng Wang Shanghai Jiao Tong University

Keywords

No keywords are indexed for this paper.