Author name cluster

Shan Zhao

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

7 papers

1 author row

AAAI Conference 2026 Conference Paper

Leveraging Image as Compressed Visual Prompt and Hierarchical Visual Knowledge for Effective Image Utilization in MLLMs

Shezheng Song
Kangcheng Ding
Shan Zhao
Shasha Li
Xiaopeng Li
Chengyu Wang
Qian Wan
Bin Ji

Multimodal Large Language Models (MLLMs) integrate text and images for complex reasoning tasks, but efficiently utilizing image remains a challenge due to redundancy and noise. Traditional methods take the entire image features as visual prompt into the MLLMs, leading to excessive visual tokens that disrupt textual information expression. Thus, recent studies treat image features as visual knowledge, storing them in the feed-forward network for retrieval when needed. These methods, completely removing images from the input, may hinder the activation of image-related knowledge. Besides, current visual knowledge focuses on fine-grained details but overlooks the hierarchical process of visual perception. As described in feature integration theory, global structure is first processed before details are integrated. Ignoring this process may lead to a fragmented visual understanding, making it difficult to capture high-level semantic relationships. To overcome these issues, we propose a novel image utilization mechanism in MLLMs. We leverage a compression-based attention mechanism to generate the compressed visual prompt, which not only mitigates the interference of excessively long visual prompts but also preserves crucial visual information necessary for activating knowledge in the MLLM. Furthermore, we extract hierarchical visual features as visual knowledge using wavelet transforms, allowing the model to capture both global structures and fine-grained details. Experiments show that our method achieves state-of-the-art performance.

PDF Details DOI

EAAI Journal 2025 Journal Article

An end-to-end semantic-guided infrared and visible registration-fusion network for advanced visual tasks

Meng Sang
Housheng Xie
Jingrui Meng
Yukuan Zhang
Junhui Qiu
Shan Zhao
Yang Yang

The integration of infrared and visible images is key to complex visual tasks, with image registration as a fundamental initial step. We explore these tasks as a unified registration-fusion process. Traditional global registration may use a multitude of feature points for alignment, achieving global optimality. However, for advanced tasks like pose recognition, object tracking, and person re-identification, the relevant semantic information may not align accurately, impacting fusion effectiveness and task performance. To address this issue, we propose a Semantic-Aware on-Demand registration-fusion network. 1) We design the Semantic-Aware Module, employing Grounding DINO and Segment Anything to capture semantic regions of interest needed for advanced tasks. 2) Combining with the Semantic-Aware Module, we design a Hierarchical Orientation Line operator and a Deep Hybrid Matching to ensure precise feature matching within semantic regions of interest. 3) To enhance fusion visual effects, a novel image fusion module is designed to facilitate high-quality image fusion within these regions. This method is versatile and applicable to a wide range of advanced visual tasks. We compare this method with six image matching and nine image fusion methods, underscoring its efficacy in advanced visual tasks. The experimental results indicate that both the registration module and the fusion module achieve optimal performance individually. When it comes to joint tasks of registration and fusion in visual tasks, our method similarly exhibits the best performance in the 6 × 10 joint tasks.

Details DOI

AAAI Conference 2024 Conference Paper

A Dual-Way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking

Shezheng Song
Shan Zhao
Chengyu Wang
Tianwei Yan
Shasha Li
Xiaoguang Mao
Meng Wang

Multimodal Entity Linking (MEL) aims at linking ambiguous mentions with multimodal information to entity in Knowledge Graph (KG) such as Wikipedia, which plays a key role in many applications. However, existing methods suffer from shortcomings, including modality impurity such as noise in raw image and ambiguous textual entity representation, which puts obstacles to MEL. We formulate multimodal entity linking as a neural text matching problem where each multimodal information (text and image) is treated as a query, and the model learns the mapping from each query to the relevant entity from candidate entities. This paper introduces a dual-way enhanced (DWE) framework for MEL: (1) our model refines queries with multimodal data and addresses semantic gaps using cross-modal enhancers between text and image information. Besides, DWE innovatively leverages fine-grained image attributes, including facial characteristic and scene feature, to enhance and refine visual features. (2)By using Wikipedia descriptions, DWE enriches entity semantics and obtains more comprehensive textual representation, which reduces between textual representation and the entities in KG. Extensive experiments on three public benchmarks demonstrate that our method achieves state-of-the-art (SOTA) performance, indicating the superiority of our model. The code is released on https://github.com/season1blue/DWE.

PDF Details DOI

AAAI Conference 2024 Conference Paper

Text-Based Occluded Person Re-identification via Multi-Granularity Contrastive Consistency Learning

Xinyi Wu
Wentao Ma
Dan Guo
Tongqing Zhou
Shan Zhao
Zhiping Cai

Text-based Person Re-identification (T-ReID), which aims at retrieving a specific pedestrian image from a collection of images via text-based information, has received significant attention. However, previous research has overlooked a challenging yet practical form of T-ReID: dealing with image galleries mixed with occluded and inconsistent personal visuals, instead of ideal visuals with a full-body and clear view. Its major challenges lay in the insufficiency of benchmark datasets and the enlarged semantic gap incurred by arbitrary occlusions and modality gap between text description and visual representation of the target person. To alleviate these issues, we first design an Occlusion Generator (OGor) for the automatic generation of artificial occluded images from generic surveillance images. Then, a fine-granularity token selection mechanism is proposed to minimize the negative impact of occlusion for robust feature learning, and a novel multi-granularity contrastive consistency alignment framework is designed to leverage intra-/inter-granularity of visual-text representations for semantic alignment of occluded visuals and query texts. Experimental results demonstrate that our method exhibits superior performance. We believe this work could inspire the community to investigate more dedicated designs for implementing T-ReID in real-world scenarios. The source code is available at https://github.com/littlexinyi/MGCC.

PDF Details DOI

AAAI Conference 2023 Conference Paper

MCL: Multi-Granularity Contrastive Learning Framework for Chinese NER

Shan Zhao
Chengyu Wang
Minghao Hu
Tianwei Yan
Meng Wang

Recently, researchers have applied the word-character lattice framework to integrated word information, which has become very popular for Chinese named entity recognition (NER). However, prior approaches fuse word information by different variants of encoders such as Lattice LSTM or Flat-Lattice Transformer, but are still not data-efficient indeed to fully grasp the depth interaction of cross-granularity and important word information from the lexicon. In this paper, we go beyond the typical lattice structure and propose a novel Multi-Granularity Contrastive Learning framework (MCL), that aims to optimize the inter-granularity distribution distance and emphasize the critical matched words in the lexicon. By carefully combining cross-granularity contrastive learning and bi-granularity contrastive learning, the network can explicitly leverage lexicon information on the initial lattice structure, and further provide more dense interactions of across-granularity, thus significantly improving model performance. Experiments on four Chinese NER datasets show that MCL obtains state-of-the-art results while considering model efficiency. The source code of the proposed method is publicly available at https://github.com/zs50910/MCL

PDF Details DOI

AAAI Conference 2021 Conference Paper

Dynamic Modeling Cross- and Self-Lattice Attention Network for Chinese NER

Shan Zhao
Minghao Hu
Zhiping Cai
Haiwen Chen
Fang Liu

Word-character lattice models have been proved to be effective for Chinese named entity recognition (NER), in which word boundary information is fused into character sequences for enhancing character representations. However, prior approaches have only used simple methods such as feature concatenation or position encoding to integrate word-character lattice information, but fail to capture fine-grained correlations in word-character spaces. In this paper, we propose DC- SAN, a Dynamic Cross- and Self-lattice Attention Network that aims to model dense interactions over word-character lattice structure for Chinese NER. By carefully combining cross-lattice and self-lattice attention modules with gated word-character semantic fusion unit, the network can explicitly capture fine-grained correlations across different spaces (e. g. , word-to-character and character-to-character), thus significantly improving model performance. Experiments on four Chinese NER datasets show that DCSAN obtains stateof-the-art results as well as efficiency compared to several competitive approaches.

PDF Details

IJCAI Conference 2020 Conference Paper

Modeling Dense Cross-Modal Interactions for Joint Entity-Relation Extraction

Shan Zhao
Minghao Hu
Zhiping Cai
Fang Liu

Joint extraction of entities and their relations benefits from the close interaction between named entities and their relation information. Therefore, how to effectively model such cross-modal interactions is critical for the final performance. Previous works have used simple methods such as label-feature concatenation to perform coarse-grained semantic fusion among cross-modal instances, but fail to capture fine-grained correlations over token and label spaces, resulting in insufficient interactions. In this paper, we propose a deep Cross-Modal Attention Network (CMAN) for joint entity and relation extraction. The network is carefully constructed by stacking multiple attention units in depth to fully model dense interactions over token-label spaces, in which two basic attention units are proposed to explicitly capture fine-grained correlations across different modalities (e. g. , token-to-token and labelto-token). Experiment results on CoNLL04 dataset show that our model obtains state-of-the-art results by achieving 90. 62% F1 on entity recognition and 72. 97% F1 on relation classification. In ADE dataset, our model surpasses existing approaches by more than 1. 9% F1 on relation classification. Extensive analyses further confirm the effectiveness of our approach.

PDF Details DOI