Author name cluster

Siying Wu

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

3 papers

1 author row

AIIM Journal 2025 Journal Article

MMSupcon: An image fusion-based multi-modal supervised contrastive method for brain tumor diagnosis

Haoyu Wang
Jing Zhang
Siying Wu
Haoran Wei
Xun Chen
Yunwei Ou
Xiaoyan Sun

The diagnosis of brain tumors is pivotal for effective treatment, with MRI serving as a commonly used non-invasive diagnostic modality in clinical practices. Fundamentally, brain tumor diagnosis is a type of pattern recognition task that requires the integration of information from multi-modal MRI images. However, existing fusion strategies are hindered by the scarcity of multi-modal imaging samples. In this paper, we propose a new training paradigm tailored for the scenario of multi-modal imaging in brain tumor diagnosis, called multi-modal supervised contrastive learning method (MMSupcon). This method significantly enhances diagnostic accuracy through two key components: multi-modal medical image fusion and multi-modal supervised contrastive loss. First, the fusion component integrates complementary imaging modalities to generate information-rich samples. Second, by introducing fused samples to guide original samples in learning feature consistency or inconsistency among classes, our loss component effectively preserves the integrity of cross-modal information while maintaining the distinctiveness of individual modalities. Finally, MMSupcon is validated on a real-world brain tumor dataset collected from Beijing Tiantan Hospital, achieving state-of-the-art performance. Furthermore, additional experiments on two public BraTS glioma classification datasets also demonstrate our substantial performance improvements. The source code is released at https: //github. com/hywang02/MMSupcon.

Details DOI

NeurIPS Conference 2024 Conference Paper

Visual Perception by Large Language Model’s Weights

Feipeng Ma
Hongwei Xue
Yizhou Zhou
Guangting Wang
Fengyun Rao
Shilin Yan
Yueyi Zhang
Siying Wu

Existing Multimodal Large Language Models (MLLMs) follow the paradigm that perceives visual information by aligning visual features with the input space of Large Language Models (LLMs) and concatenating visual tokens with text tokens to form a unified sequence input for LLMs. These methods demonstrate promising results on various vision-language tasks but are limited by the high computational effort due to the extended input sequence resulting from the involvement of visual tokens. In this paper, instead of input space alignment, we propose a novel parameter space alignment paradigm that represents visual information as model weights. For each input image, we use a vision encoder to extract visual features, convert features into perceptual weights, and merge the perceptual weights with LLM's weights. In this way, the input of LLM does not require visual tokens, which reduces the length of the input sequence and greatly improves efficiency. Following this paradigm, we propose VLoRA with the perceptual weights generator. The perceptual weights generator is designed to convert visual features to perceptual weights with low-rank property, exhibiting a form similar to LoRA. The experimental results show that our VLoRA achieves comparable performance on various benchmarks for MLLMs, while significantly reducing the computational costs for both training and inference. Code and models are released at \url{https: //github. com/FeipengMa6/VLoRA}.

PDF Details DOI

IJCAI Conference 2019 Conference Paper

Densely Supervised Hierarchical Policy-Value Network for Image Paragraph Generation

Siying Wu
Zheng-Jun Zha
Zilei Wang
Houqiang Li
Feng Wu

Image paragraph generation aims to describe an image with a paragraph in natural language. Compared to image captioning with a single sentence, paragraph generation provides more expressive and fine-grained description for storytelling. Existing approaches mainly optimize paragraph generator towards minimizing word-wise cross entropy loss, which neglects linguistic hierarchy of paragraph and results in ``sparse" supervision for generator learning. In this paper, we propose a novel Densely Supervised Hierarchical Policy-Value (DHPV) network for effective paragraph generation. We design new hierarchical supervisions consisting of hierarchical rewards and values at both sentence and word levels. The joint exploration of hierarchical rewards and values provides dense supervision cues for learning effective paragraph generator. We propose a new hierarchical policy-value architecture which exploits compositionality at token-to-token and sentence-to-sentence levels simultaneously and can preserve the semantic and syntactic constituent integrity. Extensive experiments on the Stanford image-paragraph benchmark have demonstrated the effectiveness of the proposed DHPV approach with performance improvements over multiple state-of-the-art methods.

PDF Details