Author name cluster

Fanqing Meng

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

6 papers

2 author rows

JBHI Journal 2026 Journal Article

Medical Knowledge-Driven Contrastive Learning for Similar Patient Retrieval

Fanqing Meng
Chong Feng
Ge Shi
Xia Liu
Bo Wang
Kaiyuan Zhang
Yan Zhuang

Similar patient retrieval is a fundamental task in medical informatics, aiming to identify patients with similar clinical characteristics to assist in diagnosis and treatment plan recommendation. While traditional methods relying on lexical features or medical ontologies often fail to capture implicit semantic relationships, recent advancements in dense retrieval methods powered by deep learning have shown promise yet face challenges in adapting to specific tasks such as similar patient retrieval. To address these limitations, we propose a medical knowledge-driven contrastive learning approach to enhance the representation capacity of general-purpose embedding models for medical text. Specifically, our approach introduces a novel negative sampling strategy leveraging International Classification of Diseases (ICD) codes to identify hard negatives. However, due to data imbalance issues, this method struggles to adequately mine negative examples. To overcome this limitation, we develop an external knowledge-based negative sampling method that incorporates both statistical and ambiguous knowledge, thereby enhancing the model's ability to differentiate between fine-grained medical conditions and complex clinical scenarios. We then integrate these methods into a contrastive learning framework to train more robust patient representations. Extensive experiments on real-world medical datasets show that our proposed method achieves significant improvements over existing state-of-the-art baseline models.

Details DOI

ICLR Conference 2025 Conference Paper

MMIU: Multimodal Multi-image Understanding for Evaluating Large Vision-Language Models

Fanqing Meng
Jin Wang
Chuanhao Li 0001
Quanfeng Lu
Hao Tian 0006
Tianshuo Yang
Jiaqi Liao
Xizhou Zhu

The capability to process multiple images is crucial for Large Vision-Language Models (LVLMs) to develop a more thorough and nuanced understanding of a scene. Recent multi-image LVLMs have begun to address this need. However, their evaluation has not kept pace with their development. To fill this gap, we introduce the Multimodal Multi-image Understanding (MMIU) benchmark, a comprehensive evaluation suite designed to assess LVLMs across a wide range of multi-image tasks. MMIU encompasses 7 types of multi-image relationships, 52 tasks, 77K images, and 11K meticulously curated multiple-choice questions, making it the most extensive benchmark of its kind. Our evaluation of nearly 30 popular LVLMs, including both open-source and proprietary models, reveals significant challenges in multi-image comprehension, particularly in tasks involving spatial understanding. Even the most advanced models, such as GPT-4o, achieve only 55.7\% accuracy on MMIU. Through multi-faceted analytical experiments, we identify key performance gaps and limitations, providing valuable insights for future model and data improvements. We aim for MMIU to advance the frontier of LVLM research and development. We release the data and code at https://github.com/MMIUBenchmark/MMIU.

Details

ICML Conference 2025 Conference Paper

Towards World Simulator: Crafting Physical Commonsense-Based Benchmark for Video Generation

Fanqing Meng
Jiaqi Liao
Xinyu Tan
Quanfeng Lu
Wenqi Shao
Kaipeng Zhang
Yu Cheng 0001
Dianqi Li

Text-to-video (T2V) models like Sora have made significant strides in visualizing complex prompts, which is increasingly viewed as a promising path towards constructing the universal world simulator. Cognitive psychologists believe that the foundation for achieving this goal is the ability to understand intuitive physics. However, the capacity of these models to accurately represent intuitive physics remains largely unexplored. To bridge this gap, we introduce PhyGenBench, a comprehensive Phy sics Gen eration Ben chmark designed to evaluate physical commonsense correctness in T2V generation. PhyGenBench comprises 160 carefully crafted prompts across 27 distinct physical laws, spanning four fundamental domains, which could comprehensively assesses models’ understanding of physical commonsense. Alongside PhyGenBench, we propose a novel evaluation framework called PhyGenEval. This framework employs a hierarchical evaluation structure utilizing appropriate advanced vision-language models and large language models to assess physical commonsense. Through PhyGenBench and PhyGenEval, we can conduct large-scale automated assessments of T2V models’ understanding of physical commonsense, which align closely with human feedback. Our evaluation results and in-depth analysis demonstrate that current models struggle to generate videos that comply with physical commonsense. Moreover, simply scaling up models or employing prompt engineering techniques is insufficient to fully address the challenges presented by PhyGenBench (e. g. , dynamic scenarios). We hope this study will inspire the community to prioritize the learning of physical commonsense in these models beyond entertainment applications. We will release the data and codes at https: //github. com/OpenGVLab/PhyGenBench

Details

NeurIPS Conference 2025 Conference Paper

VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models

Xiangdong Zhang
Jiaqi Liao
Shaofeng Zhang
Fanqing Meng
Xiangpeng Wan
Junchi Yan
Yu Cheng

Recent advancements in text-to-video (T2V) diffusion models have enabled high-fidelity and realistic video synthesis. However, current T2V models often struggle to generate physically plausible content due to their limited inherent ability to accurately understand physics. We found that while the representations within T2V models possess some capacity for physics understanding, they lag significantly behind those from recent video self-supervised learning methods. To this end, we propose a novel framework called {VideoREPA}, which distills physics understanding capability from video understanding foundation models into T2V models by aligning token-level relations. This closes the physics understanding gap and enables more physics-plausible generation. Specifically, we introduce the {Token Relation Distillation (TRD) loss}, leveraging spatio-temporal alignment to provide soft guidance suitable for finetuning powerful pre-trained T2V models—a critical departure from prior representation alignment (REPA) methods. To our knowledge, VideoREPA is the first REPA method designed for finetuning T2V models and specifically for injecting physical knowledge. Empirical evaluations show that VideoREPA substantially enhances the physics commonsense of baseline method, CogVideoX, achieving significant improvement on relevant benchmarks and demonstrating a strong capacity for generating videos consistent with intuitive physics. Code and more video results are available at https: //videorepa. github. io/.

PDF Details

ICML Conference 2024 Conference Paper

MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI

Kaining Ying
Fanqing Meng
Jin Wang
Zhiqian Li
Han Lin
Yue Yang
Hao Zhang 0117
Wenbo Zhang 0009

Large Vision-Language Models (LVLMs) show significant strides in general-propose multimodal applications such as visual dialogue and embodied navigation. However, existing multimodal evaluation benchmarks cover a limited number of multimodal tasks testing rudimentary capabilities, falling short in tracking LVLM development. In this study, we present MMT-Bench, a comprehensive benchmark designed to assess LVLMs across massive multimodal tasks requiring expert knowledge and deliberate visual recognition, localization, and reasoning. MMT-Bench comprises $31, 325$ meticulously curated multi-choice visual questions from various multimodal scenarios such as vehicle driving and embodied navigation, covering $32$ core meta-tasks and $162$ subtasks in multimodal understanding. Due to its extensive task coverage, MMT-Bench enables the evaluation of LVLMs using a task map, facilitating the discovery of in- and out-of-domain tasks. Evaluation results involving $20$ publicly available LVLMs such as the proprietary GeminiProVision model, underscore the significant challenges posed by MMT-Bench. We anticipate that MMT-Bench will inspire the community to develop next-generation multimodal foundation models aimed at achieving general-purpose multimodal intelligence.

Details

NeurIPS Conference 2023 Conference Paper

Foundation Model is Efficient Multimodal Multitask Model Selector

Fanqing Meng
Wenqi Shao
zhanglin peng
Chonghe Jiang
Kaipeng Zhang
Yu Qiao
Ping Luo

This paper investigates an under-explored but important problem: given a collection of pre-trained neural networks, predicting their performance on each multi-modal task without fine-tuning them, such as image recognition, referring, captioning, visual question answering, and text question answering. A brute-force approach is to finetune all models on all target datasets, bringing high computational costs. Although recent-advanced approaches employed lightweight metrics to measure models’ transferability, they often depend heavily on the prior knowledge of a single task, making them inapplicable in a multi-modal multi-task scenario. To tackle this issue, we propose an efficient multi-task model selector (EMMS), which employs large-scale foundation models to transform diverse label formats such as categories, texts, and bounding boxes of different downstream tasks into a unified noisy label embedding. EMMS can estimate a model’s transferability through a simple weighted linear regression, which can be efficiently solved by an alternating minimization algorithm with a convergence guarantee. Extensive experiments on 5 downstream tasks with 24 datasets show that EMMS is fast, effective, and generic enough to assess the transferability of pre-trained models, making it the first model selection method in the multi-task scenario. For instance, compared with the state- of-the-art method LogME enhanced by our label embeddings, EMMS achieves 9. 0%, 26. 3%, 20. 1%, 54. 8%, 12. 2% performance gain on image recognition, referring, captioning, visual question answering, and text question answering, while bringing 5. 13×, 6. 29×, 3. 59×, 6. 19×, and 5. 66× speedup in wall-clock time, respectively. The code is available at https: //github. com/OpenGVLab/Multitask-Model-Selector.

PDF Details