Multi-Modal Foundation Models for Computational Pathology: A Survey

Foundation models have emerged as a powerful paradigm in computational pathology (CPath), enabling scalable and generalizable analysis of histopathological images. While early developments centered on uni-modal models trained solely on visual data, recent advances have highlighted the promise of multi-modal foundation models that integrate heterogeneous data sources such as textual reports, structured domain knowledge, and molecular profiles. In this survey, we provide a comprehensive and up-to-date review of multi-modal foundation models in CPath, with a particular focus on models built upon hematoxylin and eosin (H&E) stained whole slide images (WSIs) and tile-level representations. We categorize 34 state-of-the-art multi-modal foundation models into three major paradigms: vision-language, vision-knowledge graph, and vision-gene expression. We further divide vision-language models into non-LLM-based and LLM-based approaches. Additionally, we analyze 30 available multi-modal datasets tailored for pathology, grouped into image-text pairs, instruction datasets, and image-other modality pairs. Our survey also presents a taxonomy of downstream tasks, highlights training and evaluation strategies, and identifies key challenges and future directions. We aim for this survey to serve as a valuable resource for researchers and practitioners working at the intersection of pathology and AI.

Authors

• Dong Li University of Science and Technology of China
• Guihong Wan Departments of Biostatistics and Epidemiology, Harvard T. H. Chan School of Public Health, MA, USA Department of Dermatology, Massachusetts General Hospital, Harvard Medical School, MA, USA
• Xintao Wu University of Arkansas
• Xinyu Wu Advanced Research Institute of Multidisciplinary Sciences, Beijing Institute of Technology, Beijing 100081, China
• Xiaohui Chen
• Yi He
• Zhong Chen Southern Illinois University-Carbondale
• Peter K Sorger Harvard Medical School, Harvard University
• Chen Zhao Nanjing University

Keywords

No keywords are indexed for this paper.