Arrow Research search

Author name cluster

Wei Sun

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

52 papers
2 author rows

Possible papers

52

AAAI Conference 2026 Conference Paper

Audio-Assisted Face Video Restoration with Temporal and Identity Complementary Learning

  • Yuqin Cao
  • Yixuan Gao
  • Wei Sun
  • Xiaohong Liu
  • Yulun Zhang
  • Xiongkuo Min

Face videos accompanied by audio have become integral to our daily lives, while they often suffer from complex degradations. Most face video restoration methods neglect the intrinsic correlations between visual and audio features, particularly in the mouth region. Several audio-aided face video restoration methods have been proposed, but they only focus on compression artifact removal. In this paper, we propose a General Audio-assisted face Video restoration Network (GAVN) to address various types of streaming video distortions via identity and temporal complementary learning. Specifically, GAVN first captures inter-frame temporal features in the low-resolution space to restore frames coarsely and save computational cost. Then, GAVN extracts intra-frame identity features in the high-resolution space with the assistance of audio signals and face landmarks to restore more facial details. Finally, the reconstruction module integrates temporal features and identity features to generate high-quality face videos. Experimental results demonstrate that GAVN outperforms the existing state-of-the-art methods on face video compression artifact removal, deblurring, and super-resolution.

PDF Details DOI

EAAI Journal 2026 Journal Article

Breaking multilayer perceptron limitations for traffic flow forecasting with structured patch learning

  • Wei Sun
  • Gong Wang
  • Junbo Gao
  • Chunyu Wang
  • Zihao Zhang

Transformer neural networks and multilayer perceptron (MLP) architectures are two dominant paradigms in multivariate spatiotemporal sequence modeling. While Transformers excel at capturing complex dependencies, their attention mechanisms incur high computational costs, limiting practical deployment. MLP methods are structurally simple and inference-efficient, but existing approaches typically rely on flattened modeling processes, making it difficult to capture hierarchical temporal patterns in traffic data. Moreover, the lack of structural decoupling between spatial and temporal components often leads to entangled representations, reducing overall modeling effectiveness. To address these challenges, we propose a structured, purely MLP model, termed the Structured Patch Multilayer Perceptron (SPMLP), for traffic flow forecasting. The model employs a temporal patch multilayer perceptron with overlapping sliding windows to extract hierarchical temporal features from local to global scales, and a graph-structured multilayer perceptron to capture node dependencies over a predefined spatial graph. A lightweight linear fusion integrates spatial and temporal representations into a unified spatiotemporal context. Extensive experiments on standard traffic flow benchmarks demonstrate that SPMLP outperforms strong baselines in both prediction accuracy and computational efficiency, highlighting the practicality of structured multilayer perceptron architectures for real-time traffic forecasting in intelligent transportation systems.

Details DOI

JBHI Journal 2026 Journal Article

CapsFormer: A Dual-Stream Causal-Aware Capsule-Transformer Network for EMG Signal Representation Learning

  • Pengpai Wang
  • Tiantian Xie
  • Yueying Zhou
  • Peiliang Gong
  • Wei Sun
  • Xiaorui Zhang
  • Rosa H. M. Chan

Electromyography (EMG) signals are widely applied in prosthetic control, rehabilitation training, and human-machine interaction. This places stringent requirements on gesture recognition algorithms to balance long-range temporal modeling with local pose invariance. Existing approaches typically trade off between local and global features and lack causally consistent interpretability, resulting in insufficient generalization across subjects and sessions. To address these shortcomings, we propose a novel dual-stream causal Capsule-Transformer network (CapsFormer). In the Transformer stream, we employ a “causal attention” in the self-attention mechanism to explicitly block all future information, ensuring that each time-step representation depends solely on itself and prior signals; in the Capsule stream, we leverage dynamic routing to capture local part-whole pose vectors, enhancing robustness against electrode shifts and muscle deformations. The two streams' features are then integrated in a fusion module and trained end-to-end. To validate the model's effectiveness, we evaluate it on a multi-subject dataset; results demonstrate that CapsFormer outperforms state-of-the-art models in recognition accuracy, cross-subject robustness, and interpretability. This work not only offers a new paradigm for efficient EMG signal representation but also supports causally consistent temporal signal analysis and interpretable deep learning methods, bearing significant implications for intelligent prosthetic control and human-machine interfaces.

Details DOI

EAAI Journal 2026 Journal Article

Dual-domain attentions for unmanned aerial vehicle small object detection

  • Chunmei Wang
  • Yunxiao Chang
  • Shan Xie
  • Xiaobao Yang
  • Yadong Tian
  • Wei Sun
  • Junyan Hu

Images captured by unmanned aerial vehicles (UAVs) often suffer from severe degradation in small object quality and resolution due to environmental constraints, posing significant challenges in preserving the dual-domain characteristics of spatial details and frequency components. While large-scale models attempt to address this through complex architectures, aggressive down-sampling and successive convolution operations inevitably erase fine-grained patterns that are essential for detecting small objects. To overcome these challenges, we propose a dual-domain attention mechanism for small object detection, which focuses on both spatial and frequency domains. In the spatial domain, the proposed step-free triple-attention convolution (SFTAConv) reduces information loss during feature propagation by combining spatial–channel interactions and a lossless space-to-depth transform, thereby enhancing subtle object patterns while suppressing background interference. In the frequency domain, the frequency-domain hybrid attention (FD-HAT) jointly recalibrates high- and low-frequency components, moving beyond single-domain recalibration to recover discriminative representations of occluded or blurred small objects. Additionally, a classification-assisted localization (CAL) branch with classification-guided localization further refines detection accuracy. After extensive experiments conducted on the vision meets drone 2019 object detection (VisDrone2019Det), dataset for object detection in aerial (DOTA), and pascal visual object classes (PASCAL VOC) datasets, the results demonstrate that our model achieved the significant gains of 2. 2%, 1. 7%, 5. 3% at A P s metric on three datasets, respectively, and being competitive with the state-of-the-art (SOTA) detectors.

Details DOI

AAAI Conference 2026 Conference Paper

FedBRICK: Structural Bias Aware Heterogeneous Foundation Model Federated Tuning

  • Yuhang Zhang
  • Xianda Wang
  • Wei Sun
  • Jiaxuan Chen
  • Fangxin Wang

Model-heterogeneous federated tuning (MHFT) enables the privacy-preserving fine-tuning of foundation models in heterogeneous systems by allowing clients and the server to adopt different model architectures. Depth partial training—where each client updates only a subset of the model's layers—alleviates system heterogeneity but exacerbates client drift, which stems from clients optimizing different objectives and therefore degrades overall performance. Beyond the well-known statistical bias—where non-IID data leads to client drift—we identify a structural bias arising from clients deploying only partial layers of the global model, which serves as an important cause of drift. We further provide a theoretical analysis showing that the possible range of structural bias expands linearly with the number of missing layers. To counter this effect, we introduce FedBRICK (Federated Bias Recovery via Inserted Calibrative Kernels), which inserts tiny BRICKs into each client’s subnetwork. We employ a dual-end layer-wise distillation scheme to train these blocks using both client-side local data and a small public proxy set on the server. This design effectively mitigates the structural bias caused by layer dropping, reduces client drift, and remains practical for storage-constrained devices. Extensive experiments on federated learning benchmarks confirm that FedBRICK delivers up to a 5% average accuracy gain while requiring no more than 1.44% extra storage per client.

PDF Details DOI

EAAI Journal 2026 Journal Article

Progressive boundary optimisation with cross-knowledge enhancement for arbitrary-shape text detection

  • Wei Sun
  • Yaqi Wang
  • Qianzhou Wang
  • Xianguang Kong
  • Yanning Zhang

The detection of text instances with arbitrary styles remains a major source of errors in scene text understanding. The multi-lingual, multi-oriented and multi-scale problems greatly reduce text salience. While recent studies have proposed various learning frameworks to tackle this issue, many rely on complex post-processing and struggle to handle extreme scale variations. Towards this end, we explore a unified coarse-to-fine framework via multi-scale cross-knowledge learning for arbitrary-shape text detection. Unlike previous methods that model feature point correlations in a holistic manner, our approach adaptively selects a small set of key sampling points around the reference. This not only reduces computational overhead across varying scales but also mitigates interference from background noise. Moreover, incorporating multi-scale information under semantic priors further strengthens the reliability of dependency modelling. Extensive experiments on widely used benchmarks demonstrate that our method, guided by cross-knowledge and adaptive attention, achieves competitive performance. Specifically, it attains F-measure scores of 92. 8% on MSRA-TD500, 87. 1% on MSRA-TD500, and 90. 5% on Total-Text.

Details DOI

AAAI Conference 2026 Conference Paper

Self-Enhanced Image Clustering with Cross-Modal Semantic Consistency

  • Zihan Li
  • Wei Sun
  • Jing Hu
  • Jianhua Yin
  • Xing Wang
  • Erwei Yin
  • Jianlong Wu

While large language-image pre-trained models like CLIP offer powerful generic features for image clustering, existing methods typically freeze the encoder. This creates a fundamental mismatch between the model's task-agnostic representations and the demands of a specific clustering task, imposing a ceiling on performance. To break this ceiling, we propose a self-enhanced framework based on cross-modal semantic consistency for efficient image clustering. Our framework first builds a strong foundation via Cross-Modal Semantic Consistency and then specializes the encoder through Self-Enhancement. In the first stage, we focus on Cross-Modal Semantic Consistency. By mining consistency between generated image-text pairs at the instance, cluster assignment, and cluster center levels, we train lightweight clustering heads to align with the rich semantics of the pre-trained model. This alignment process is bolstered by a novel method for generating higher-quality cluster centers and a dynamic balancing regularizer to ensure well-distributed assignments. In the second stage, we introduce a Self-Enhanced fine-tuning strategy. The well-aligned model from the first stage acts as a reliable pseudo-label generator. These self-generated supervisory signals are then used to feed back the efficient, joint optimization of the vision encoder and clustering heads, unlocking their full potential. Extensive experiments on six mainstream datasets show that our method outperforms existing deep clustering methods by significant margins. Notably, our ViT-B/32 model already matches or even surpasses the accuracy of state-of-the-art methods built upon the far larger ViT-L/14.

PDF Details DOI

AAAI Conference 2026 Conference Paper

VQAThinker: Exploring Generalizable and Explainable Video Quality Assessment via Reinforcement Learning

  • Linhan Cao
  • Wei Sun
  • Weixia Zhang
  • Xiangyang Zhu
  • Jun Jia
  • Kaiwei Zhang
  • Dandan Zhu
  • Guangtao Zhai

Video quality assessment (VQA) aims to objectively quantify perceptual quality degradation in alignment with human visual perception. Despite recent advances, existing VQA models still suffer from two critical limitations: poor generalization to out-of-distribution (OOD) videos and limited explainability, which restrict their applicability in real-world scenarios. To address these challenges, we propose VQAThinker, a reasoning-based VQA framework that leverages large multimodal models (LMMs) with reinforcement learning to jointly model video quality understanding and scoring, emulating human perceptual decision-making. Specifically, we adopt group relative policy optimization (GRPO), a rule-guided reinforcement learning algorithm that enables reasoning over video quality under score-level supervision, and introduce three VQA-specific rewards: (1) a bell-shaped regression reward that increases rapidly as the prediction error decreases and becomes progressively less sensitive near the ground truth; (2) a pairwise ranking reward that guides the model to correctly determine the relative quality between video pairs; and (3) a temporal consistency reward that encourages the model to prefer temporally coherent videos over their perturbed counterparts. Extensive experiments demonstrate that VQAThinker achieves state-of-the-art performance on both in-domain and OOD VQA benchmarks, showing strong generalization for video quality scoring. Furthermore, evaluations on video quality understanding tasks validate its superiority in distortion attribution and quality description compared to existing explainable VQA models and LMMs. These findings demonstrate that reinforcement learning offers an effective pathway toward building generalizable and explainable VQA models solely with score-level supervision.

PDF Details DOI

EAAI Journal 2025 Journal Article

A looped-type functional for non-fragile fuzzy sampled-data control of doubly fed induction generator-based wind energy conversion systems with failures

  • Xiaoqing Li
  • Kaibo Shi
  • Liang Han
  • Jun Cheng
  • Wei Sun
  • Zhinan Peng

In our investigation, by integrating the fuzzy modeling method and looped-type Lyapunov functional in tandem with free-matrix-based integral inequality technique, the nonfragile fault tolerant fuzzy sampled-data control (FTFSDC) problem for doubly fed induction generator (DFIG)-based wind energy conversion systems (WECSs) that are prone to semi-Markovian jump-type actuator failures (SMJAFs) has been examined. First and foremost, fuzzy modeling method, which well-suited for addressing high nonlinearity issues, is exploited to improve the imprecision of DFIG-based WECSs. Subsequently, to delineate the stochastic actuator failures (SAFs), the FTFSDC framework is reconstructed with semi-Markovian jump faulty coefficients undergoing partially unknown transition rates (PUTRs), which has stronger modeling precisely and exhibits the practical scenarios more accurately. More concretely, the PUTRs considered in this article including completely unknown elements and uncertain but bounded elements as two special situations. Thirdly, in a departure from the anterior sampled-data control (SDC) mechanisms, the memory signal τ is subtly embedded into the communication channel, and the sampling patterns in conjunction with the control actions are performed in a sawtooth-characteristic manner. Following that, by adequately capturing the characteristic information of the entire sampling intervals [ t k, t k + 1 ) and [ t k − τ, t k + 1 − τ ) simultaneously, an augmented dual-sided looped-type Lyapunov functional is put forward. The superiority of the newly constructed Lyapunov functional is that more sawtooth structure information of the actual sampling patterns is developed and some symmetric matrices involved are no longer restricted to positive definite, which can significantly enhance the design freedom and flexibility. Afterwards, with the assistance of mathematical deduction and stochastic analysis technique, several certain sufficient conditions such that the underlying plant is asymptotically stable are lastly precisely inferred in the shape of linear matrix inequalities (LMIs). In conclusion, a corresponding numerical simulation is conducted to thoroughly substantiate the efficacy and feasibility of the theoretical outcomes.

Details DOI

ICLR Conference 2025 Conference Paper

API Pack: A Massive Multi-Programming Language Dataset for API Call Generation

  • Zhen Guo
  • Adriana Meza Soria
  • Wei Sun
  • Yikang Shen
  • Rameswar Panda

We introduce API Pack, a massive multi-programming language dataset containing over one million instruction-API calls for improving the API call generation capabilities of large language models. Our evaluation highlights three key findings: First, fine-tuning on API Pack enables open-source models to outperform GPT-3.5 and GPT-4 in generating code for entirely new API calls. We show this by fine-tuning CodeLlama-13B on 20,000 Python instances from API Pack. Second, fine-tuning on a large dataset in one language, combined with smaller datasets from others, improves API generation accuracy across multiple languages. Third, we confirm the benefits of larger datasets for API generalization, as increasing fine-tuning data to one million instances enhances generalization to new APIs. To support further research, we open-source the API Pack dataset, trained model, and code at https://github.com/zguo0525/API-Pack.

Details

NeurIPS Conference 2025 Conference Paper

Causal LLM Routing: End-to-End Regret Minimization from Observational Data

  • Asterios Tsiourvas
  • Wei Sun
  • Georgia Perakis

LLM routing aims to select the most appropriate model for each query, balancing competing performance metrics such as accuracy and cost across a pool of language models. Prior approaches typically adopt a decoupled strategy, where the metrics are first predicted and the model is then selected based on these estimates. This setup is prone to compounding errors and often relies on full-feedback data, where each query is evaluated by all candidate models, which is costly to obtain and maintain in practice. In contrast, we learn from observational data, which records only the outcome of the model actually deployed. We propose a causal end-to-end framework that learns routing policies by minimizing decision-making regret from observational data. To enable efficient optimization, we introduce two theoretically grounded surrogate objectives: a classification-based upper bound, and a softmax-weighted regret approximation shown to recover the optimal policy at convergence. We further extend our framework to handle heterogeneous cost preferences via an interval-conditioned architecture. Experiments on public benchmarks show that our method outperforms existing baselines, achieving state-of-the-art performance across different embedding models.

PDF Details

EAAI Journal 2025 Journal Article

Decoupled Graph Spatial-Temporal Transformer Networks for traffic flow forecasting

  • Wei Sun
  • Rongzhang Cheng
  • Yingqi Jiao
  • Junbo Gao

Recent research has extensively applied the Transformer architecture to traffic flow prediction, proposing various solutions to effectively capture the nonlinear spatial-temporal correlations of traffic data. To better model traffic flow data, some works have highlighted the need to simultaneously consider the spatial-temporal heterogeneity during the learning of spatial-temporal correlations of traffic flow data. However, none of the existing methods have successfully achieved the goal of effectively capturing spatial-temporal heterogeneity while learning spatial-temporal correlations. To address these challenges, we propose a model for traffic flow prediction, named Decoupled Graph Spatial-Temporal Transformer Networks (DSTTN). The core of this model is a Spatial-Temporal Decoupled Representation Learning module designed to decouple spatial-temporal embedding and apply it to Transformer Networks, integrating with the Multi-Head Attention Mechanism. Through the Spatial-Temporal Decoupled Representation Learning Attention Mechanism, the model effectively achieves spatial-temporal decoupled learning of input feature embedding after spatial-temporal embedding structure embedding, thereby enhancing the ability of the model to learn spatial-temporal features for representation learning. This enables the model to extract spatial-temporal correlations and model spatial-temporal heterogeneity based on the Graph Transformer model. Experiments conducted on six real-world traffic datasets demonstrate that our model outperforms twelve comparative models, validating the ability of the model to perform more accurate prediction tasks. Additionally, ablation experiments confirm the effectiveness of the proposed spatial-temporal decoupling structure based on Transformer.

Details DOI

ICRA Conference 2025 Conference Paper

Design of a Bioinspired Jumping Mechanism for Self-Takeoff of Flapping Robot

  • Erzhen Pan
  • Wei Sun
  • Wenfu Xu

Most birds in nature rely on jumping for takeoff. Flapping-Wing Robots can flap and fly like birds but require an operator to take off, which are unable to generate sufficient lift to maintain flight at a low airspeed and must accelerate to take-off speed in a short time. It poses a challenge for the design of the jumping mechanism. This study is inspired by the jump-takeoff of birds and designs a simple and lightweight jumping leg, which is capable of storing and releasing energy with only one degree of freedom. In addition, a prototype was developed and tested, with a wingspan of 2 meters and a mass of 1. 6 kilograms, accelerating to 4 m/s in 52 ms by jumping, achieving the jumping take-off from the ground.

Details

EAAI Journal 2025 Journal Article

Differential recurrent autoencoder for early fault detection of nonstationary industrial processes

  • Cheng Ji
  • Tingting Tao
  • Fangyuan Ma
  • Jingde Wang
  • Wei Sun

Artificial Intelligence (AI) has emerged as a prominent technique for industrial fault detection, where deep learning demonstrates unique advantages in modeling nonlinear and dynamic process systems. Nevertheless, the presence of stochastic factors and permissible external disturbances frequently induce nonstationary characteristic during industrial process operation, which poses significant obstacles to early fault detection, particularly in capturing time-varying statistical patterns that precede critical failures. To bridge this gap, we propose a differential recurrent autoencoder (DRAE) for early fault detection of nonstationary industrial processes. DRAE based on differential operation, aiming to optimize the update mechanism of the forget gate to enhances its capacity to capture short-term nonstationary features. To facilitate the transition of time-variant nonstationary information in memory states, differential operations are embedded into the latent layer of Long Short-Term Memory (LSTM) neural network. The integration of the differential gradient layer with LSTM units directly models transient nonstationary dynamics while retaining original sequential dependencies, thereby overcoming the stationarity assumption inherent in conventional process monitoring approaches. Combined with an unsupervised encoder-decoder structure, this framework enables automatic feature extraction through fault-discriminative reconstruction to perform early fault detection. The proposed method is validated on a nonstationary numerical process, the Tennessee Eastman (TE) process, and a real-world chemical industrial process. The experimental results demonstrate significant improvement in fault detection rate and reduced detection delays compared to baseline methods, showcasing the potential of AI in enhancing process safety management within industrial systems.

Details DOI

JBHI Journal 2025 Journal Article

DoctorPupil: A Virtual Reality System for Parkinson's Diagnosis Through Task-Evoked Pupil Response

  • Xucheng Zhang
  • Zhirong Wan
  • Jing Zhao
  • Xinjin Li
  • Anfeng Liu
  • Xiangmin Fan
  • Wei Sun
  • Feng Tian

Parkinson's Disease (PD) is one of the most critical neurodegenerative diseases, yet there is no cure for it, and the state-of-the-art treatment is to slow its progression. Thus, the earlier a patient with PD is recognized, the better he can be treated. Our project joins the research effort that aims to support early PD diagnosis by designing a Virtual Reality (VR)-based system to monitor pupil diameter patterns as new biomarkers (e. g. , Pupil Light Reflex and Task-evoked Pupil Response) and provide early warning of potential PD onset. A follow-up experiment with 55 participants shows that the accuracy of recognizing early PD from healthy controls could reach 0. 8942. Our study shows early results of a promising research direction that leverages VR-based technology to non-intrusively recognize patterns and provide alerts to early PD patients who would otherwise not know their symptoms until much later.

Details DOI

ICRA Conference 2025 Conference Paper

Doppler Former: Velocity Supervision of Raw Radar Data

  • Shuo Zhao
  • Wei Sun
  • Huadong Li
  • Zhaoying Jiang

Thanks to the high robustness of 4D millimeterwave radar in various environments, it has been widely applied in the field of autonomous driving. Recent research has increasingly focused on utilizing raw data, as a substitute for the sparse and noisy point cloud data. However, these approaches have not fully exploited the Doppler features present in the raw data. In this paper, we introduce the Doppler Former (DPF) module to efficiently extract velocity information from the target environment. DPF can be seamlessly integrated into most radar perception backbone and enhance their performance in downstream tasks. Additionally, we propose a new backbone, Fully Complex Convolutional Network (FCCN), which is more suitable for raw data. By incorporating the DPF module into FCCN, we achieved state-of-the-art (SOTA) performance on the RADIal dataset, with code available at https://github.com/coconut-zs/Fvidar-DopplerFormer.

Details

NeurIPS Conference 2025 Conference Paper

KTAE: A Model-Free Algorithm to Key-Tokens Advantage Estimation in Mathematical Reasoning

  • Wei Sun
  • Wen Yang
  • Pu Jian
  • Qianlong Du
  • Fuwei Cui
  • Shuo Ren
  • Jiajun Zhang

Recent advances have demonstrated that integrating reinforcement learning with rule-based rewards can significantly enhance the reasoning capabilities of large language models (LLMs), even without supervised fine-tuning (SFT). However, prevalent reinforcement learning algorithms such as GRPO and its variants like DAPO, suffer from a coarse granularity issue when computing the advantage. Specifically, they compute rollout-level advantages that assign identical values to every token within a sequence, failing to capture token-specific contributions. To address this limitation, we propose Key-token Advantage Estimation (KTAE)—a novel algorithm that estimates fine-grained, token-level advantages without introducing additional models. KTAE leverages the correctness of sampled rollouts and applies statistical analysis to quantify the importance of individual tokens within a sequence to the final outcome. This quantified token-level importance is then combined with the rollout-level advantage to obtain a more fine-grained token-level advantage estimation. Empirical results show that models trained with GRPO+KTAE and DAPO+KTAE outperform baseline methods across five mathematical reasoning benchmarks. Notably, they achieve higher accuracy with shorter responses and even surpass R1-Distill-Qwen-1. 5B using the same base model.

PDF Details

ICRA Conference 2025 Conference Paper

LamPro: Multi-Prototype Representation Learning for Enhanced Visual Pattern Recognition

  • Ji Qi
  • Wei Sun
  • Qihe Huang
  • Zhengyang Zhou
  • Yang Wang 0015

Visual pattern recognition usually plays important roles in robotics and automation society where the pattern recognition relies on representation learning. Existing representation learning often neglects two important issues, the diversity of intra-class representation and under-exploited label utilization, especially the negative feedback during training process. Fortunately, prototype learning potentially raises label utilization and encourages intra-class diversity. In this paper, we investigate the intra-class diversity and effective updates in prototype learning for enhanced visual pattern recognition. Specifically, we propose a Label-aware multi-Prototype learning, LamPro, by incorporating the label awareness into both prototype formation and update to improve the representation quality. Firstly, we design a supervised contrastive learning to achieve class-discriminative representations. Secondly, we randomly initialize multiple prototypes and update the nearest prototype upon the arrival of instance, to preserve intra-class diversity. Thirdly, we propose a novel Label-guided Adaptive Updating. We separate the prototype updates from the representation optimization and exploit the label indexes to directly implement the prediction feedback. To correct the model optimization directions, we identify the negative feedback, and correct the prototype updates via queries of labels. Finally, we design a memory-based counter to alternately update these deviated prototypes. Experiments verify the effectiveness of our label-aware and joint multi-prototype updating strategies.

Details

ICLR Conference 2025 Conference Paper

Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models

  • Linh Tran
  • Wei Sun
  • Stacy Patterson
  • Ana L. Milanova

Multimodal Large Language Models (LLMs) are pivotal in revolutionizing customer support and operations by integrating multiple modalities such as text, images, and audio. Federated Prompt Learning (FPL) is a recently proposed approach that combines pre-trained multimodal LLMs such as vision-language models with federated learning to create personalized, privacy-preserving AI systems. However, balancing the competing goals of personalization, generalization, and privacy remains a significant challenge. Over-personalization can lead to overfitting, reducing generalizability, while stringent privacy measures, such as differential privacy, can hinder both personalization and generalization. In this paper, we propose a Differentially Private Federated Prompt Learning (DP-FPL) approach to tackle this challenge by leveraging a low-rank factorization scheme to capture generalization while maintaining a residual term that preserves expressiveness for personalization. To ensure privacy, we introduce a novel method where we apply local differential privacy to the two low-rank components of the local prompt, and global differential privacy to the global prompt. Our approach mitigates the impact of privacy noise on the model performance while balancing the tradeoff between personalization and generalization. Extensive experiments demonstrate the effectiveness of our approach over other benchmarks.

Details

ICLR Conference 2025 Conference Paper

Semi-Supervised Vision-Centric 3D Occupancy World Model for Autonomous Driving

  • Xiang Li 0205
  • Pengfei Li 0007
  • Yupeng Zheng
  • Wei Sun
  • Yan Wang
  • Yilun Chen

Understanding world dynamics is crucial for planning in autonomous driving. Recent methods attempt to achieve this by learning a 3D occupancy world model that forecasts future surrounding scenes based on current observation. However, 3D occupancy labels are still required to produce promising results. Considering the high annotation cost for 3D outdoor scenes, we propose a semi-supervised vision-centric 3D occupancy world model, **PreWorld**, to leverage the potential of 2D labels through a novel two-stage training paradigm: the self-supervised pre-training stage and the fully-supervised fine-tuning stage. Specifically, during the pre-training stage, we utilize an attribute projection head to generate different attribute fields of a scene (e.g., RGB, density, semantic), thus enabling temporal supervision from 2D labels via volume rendering techniques. Furthermore, we introduce a simple yet effective state-conditioned forecasting module to recursively forecast future occupancy and ego trajectory in a direct manner. Extensive experiments on the nuScenes dataset validate the effectiveness and scalability of our method, and demonstrate that PreWorld achieves competitive performance across 3D occupancy prediction, 4D occupancy forecasting and motion planning tasks.

Details

EAAI Journal 2025 Journal Article

Ship re-identification in foggy weather: A two-branch network with dynamic feature enhancement and dual attention

  • Wei Sun
  • Fei Guan
  • Xiaorui Zhang
  • Xinyi Shen
  • Kaibo Wang

Most existing ship re-identification methods are only applicable under sunny days. However, the presence of fog makes ship re-identification difficult. Therefore, this study proposes a two-branch network with dynamic feature enhancement and dual attention, which includes the defogging and re-identification branch, employing an end-to-end training approach to simultaneously learn defogging and re-identification tasks. To address issues related to blurring images and changes in ship appearance caused by fog, this study proposes a dynamic feature enhancement module, utilizing the deformable group convolution to adaptively adjust the shapes and positions of kernels. The adjustment enables better matching with ship actual outlines and size changes, thereby accurately locating salient regions in ship images. To further improve the network’s ability in capturing ship details in foggy weather and recover clear and fog-free images, a dual attention module fusing multiple pooling operations and using convolution kernels of different sizes is proposed, which strengthens the network’s attention to capture ship characteristics in different channels and spaces. To enhance the discrimination of learned ship vital attributes, this study introduces the adaptive sparse pair loss, dynamically adjusting the network’s attention to positive and negative sample pairs with different attributes. In addition, a new dataset is constructed specifically for foggy ship re-identification. The atmospheric scattering model is employed for the synthesis of foggy ship images, effectively addressing the challenge associated with training and evaluating foggy ship re-identification networks. Experimental results show that the proposed method outperforms existing ship re-identification methods in foggy weather.

Details DOI

EAAI Journal 2025 Journal Article

Unsupervised multiple-target domain adaptation for bearing fault diagnosis

  • Guoli Bai
  • Tonghao Xing
  • Wei Sun
  • Huashan Chi
  • Zhidan Zhong
  • Qingchao Sun
  • Liang Sun

Data-driven bearing fault diagnosis methods have gained significant attention due to their high accuracy and minimal reliance on expert knowledge. However, models trained on the data collected under single working conditions often assume independent and identically distributed data, limiting their generalization ability in real-world scenarios where varying working conditions introduce challenges such as domain shift and unlabeled data. Unsupervised domain adaptation is an effective method for addressing these challenges by training classifiers using labeled source domain data and unlabeled target domain data. However, current domain adaptation fault diagnosis methods are typically limited to single-source and single-target scenarios. When multiple unlabeled working conditions are involved, it becomes difficult to simultaneously ensure the domain invariance and the fault relevance of features. This paper proposes a fault diagnosis method across multiple unlabeled working conditions. The method extracts features from image data transformed from time-domain signals and ensures the correlation between the extracted features and fault labels, as well as domain invariance, through the gradient reversal mechanism and a corresponding loss function. The proposed method achieves an average accuracy of 99. 89 % and 95. 93 % in multi-target domain fault diagnosis tasks on two public benchmark bearing datasets, which illustrates its effectiveness and advantages compared with the existing representative methods. The source code is available at https: //github. com/WhiteGL/DA_IRP/tree/master.

Details DOI

EAAI Journal 2025 Journal Article

Visual-tactile fusion learning for material recognition based on channel switching and dual cross-attention

  • Song Li
  • Wei Sun
  • Qiaokang Liang
  • Jian Sun
  • Hui Yang
  • YuDong Yang

Visual-tactile multimodal object recognition has attracted increasing attention, as information from different modalities can complement each other and enhance recognition performance. However, the inherent heterogeneity between vision and touch presents a key challenge for effective fusion, limiting accuracy and robustness. To address this, we propose a Multimodal Channel-Switching and Dual Cross-Attention Fusion (MCSDCF) method for integrating multisource data. The channel-switching module adaptively weights and exchanges information across modalities, enabling more discriminative and complementary feature representations. To further capture high-level semantic correlations and heterogeneous cues, we introduce a dual crossattention fusion structure that combines intra-modal self-attention with cross-modal mutual attention, reinforcing the quality of fused representations. Extensive experiments on three public benchmark datasets demonstrate the effectiveness of our MCSDCF framework, with multimodal fusion consistently outperforming unimodal baselines. Ablation studies further confirm the individual contributions of the proposed modules to the overall recognition performance.

Details DOI

ECAI Conference 2024 Conference Paper

ANTIDOTE: ArgumeNtaTIon-Driven explainable artificial intelligence fOr digiTal mEdicine

  • Cristian Cardellino
  • Theo Alkibiades Collias
  • Benjamin Molinet
  • Erwan Hain
  • Wei Sun
  • Rodrigo Agerri
  • Serena Villata
  • Elena Cabrio

The need for transparent AI systems in sensitive domains like medicine has become key. In this paper we present ANTIDOTE, a software suite proposing different tools for argumentation-driven explainable Artificial Intelligence for digital medicine. Our system offers the following functionalities: multilingual argumentative analysis for the medical domain, explanation extraction and generation of clinical diagnoses, multilingual large language models for the medical domain, and the first multilingual benchmark for medical question-answering. Experimental results demonstrate the efficacy of ANTIDOTE across different tasks, highlighting its potential as an asset in medical research and practice and fostering transparency, which is crucial for informed decision-making in healthcare.

Details

IJCAI Conference 2024 Conference Paper

DiffStega: Towards Universal Training-Free Coverless Image Steganography with Diffusion Models

  • Yiwei Yang
  • Zheyuan Liu
  • Jun Jia
  • Zhongpai Gao
  • Yunhao Li
  • Wei Sun
  • Xiaohong Liu
  • Guangtao Zhai

Traditional image steganography focuses on concealing one image within another, aiming to avoid steganalysis by unauthorized entities. Coverless image steganography (CIS) enhances imperceptibility by not using any cover image. Recent works have utilized text prompts as keys in CIS through diffusion models. However, this approach faces three challenges: invalidated when private prompt is guessed, crafting public prompts for semantic diversity, and the risk of prompt leakage during frequent transmission. To address these issues, we propose DiffStega, an innovative training-free diffusion-based CIS strategy for universal application. DiffStega uses a password-dependent reference image as an image prompt alongside the text, ensuring that only authorized parties can retrieve the hidden information. Furthermore, we develop Noise Flip technique to further secure the steganography against unauthorized decryption. To comprehensively assess our method across general CIS tasks, we create a dataset comprising various image steganography instances. Experiments indicate substantial improvements in our method over existing ones, particularly in aspects of versatility, password sensitivity, and recovery quality. Codes are available at https: //github. com/evtricks/DiffStega.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

GAIA: Rethinking Action Quality Assessment for AI-Generated Videos

  • Zijian Chen
  • Wei Sun
  • Yuan Tian
  • Jun Jia
  • Zicheng Zhang
  • Jiarui Wang
  • Ru Huang
  • Xiongkuo Min

Assessing action quality is both imperative and challenging due to its significant impact on the quality of AI-generated videos, further complicated by the inherently ambiguous nature of actions within AI-generated video (AIGV). Current action quality assessment (AQA) algorithms predominantly focus on actions from real specific scenarios and are pre-trained with normative action features, thus rendering them inapplicable in AIGVs. To address these problems, we construct GAIA, a Generic AI-generated Action dataset, by conducting a large-scale subjective evaluation from a novel causal reasoning-based perspective, resulting in 971, 244 ratings among 9, 180 video-action pairs. Based on GAIA, we evaluate a suite of popular text-to-video (T2V) models on their ability to generate visually rational actions, revealing their pros and cons on different categories of actions. We also extend GAIA as a testbed to benchmark the AQA capacity of existing automatic evaluation methods. Results show that traditional AQA methods, action-related metrics in recent T2V benchmarks, and mainstream video quality methods perform poorly with an average SRCC of 0. 454, 0. 191, and 0. 519, respectively, indicating a sizable gap between current models and human action perception patterns in AIGVs. Our findings underscore the significance of action quality as a unique perspective for studying AIGVs and can catalyze progress towards methods with enhanced capacities for AQA in AIGVs.

PDF Details DOI

EAAI Journal 2024 Journal Article

Image deraining via invertible disentangled representations

  • Xueling Chen
  • Xuan Zhou
  • Wei Sun
  • Yanning Zhang

Photos taken on rainy days usually suffer from a loss of background image information due to obstruction by rain streaks and the mist-like effect that rain creates. Recent Invertible Neural Networks (INNs) have shown promise in image deraining, as they are capable of extracting observed features without information loss for the purpose of restoration. However, it remains a challenge for INNs to directly learn the transformation from a rainy image to its rain-free version, given that rain streaks exhibit a diversity in transparency, shape, and brightness, leading to a complex distribution of lost details. To solve the problem, we design a novel Invertible Disentangling network (InvDis) to decompose the complex data distribution into simpler ones and makes the transformation much easier to learning. To be specific, InvDis separates feature channels of rain streaks and those of the polluted image details in the forward mapping. By only discarding rain channels and preserving polluted image details, in the backward mapping, InvDis focuses on generating missing details obstructed by rain streaks with a new sample from a prior distribution, and restores clean details from those polluted ones. Additionally, InvDis incorporates a channel interaction mechanism to facilitate the disentangling. It allows the rain streaks encoded in other channels to easily flow to those discarded channels, and conversely, allows the polluted image details to be directed towards the preserved channels. Experimental results on both synthetic and real-world datasets show that the InvDis not only improves the restoration quality but also has lower computational costs.

Details DOI

EAAI Journal 2024 Journal Article

Smooth fusion of multi-spectral images via total variation minimization for traffic scene semantic segmentation

  • Ying Li
  • Aiqing Fang
  • Yangming Guo
  • Wei Sun
  • Xiaobao Yang
  • Xiaodong Wang

Achieving precise semantic segmentation for traffic scenes relies on adopting multi-spectral image fusion techniques to attain high-quality images. Many existing fusion solutions often aim to enhance the similarity between the input and fusion results at the pixel intensity and texture details stage. However, this can result in smoothness issues that limit semantic segmentation performance. To address these issues, we present a smooth representation learning optimization mechanism (SFLM) that conducts image fusion on two dimensions: inter- and intra-image levels. The former overcomes over- or under-smoothing problems via the mutual information maximization between the fusion result and image samples (i. e. , negative and positive). The latter balances under and over-smoothing for fusion results by minimizing the total variation in pixel space and maximizing the total variation in gradient space based on contrast learning. In this way, the proposed method effectively overcomes the fusion quality issues, providing better feature representations for semantic segmentation in autonomous vehicles. Experimental results on four public datasets validate our method’s effectiveness, robustness, and overall superiority.

Details DOI

IROS Conference 2024 Conference Paper

Visual Timing For Sound Source Depth Estimation in the Wild

  • Wei Sun
  • Lili Qiu

Depth estimation enables a wide variety of 3D applications, such as robotics and autonomous driving. Despite significant work on various depth sensors, it is challenging to develop an all-in-one method to meet multiple basic criteria. In this paper, we propose a novel audio-visual learning scheme by integrating semantic features with physical spatial cues to boost monocular depth with only one microphone. Inspired by the flash-to-bang theory, we develop FBDepth, the first passive audio-visual depth estimation framework. It is based on the difference between the time-of-flight (ToF) of the light and the sound. We formulate sound source depth estimation as an audio-visual event localization task for collision events. To approach decimeter-level depth accuracy, we design a coarse-to-fine pipeline to push the temporary localization accuracy from event-level to millisecond-level by aligning audio-visual correspondence and manipulating optical flow. FBDepth feeds the estimated visual timestamp together with the audio clip and objects visual features to regress the source depth. We use a mobile phone to collect 3. 6K+ video clips with 24 different objects at up to 65m. FBDepth shows superior performance especially at a long range compared to monocular and stereo methods.

Details

ICML Conference 2023 Conference Paper

Learning Prescriptive ReLU Networks

  • Wei Sun
  • Asterios Tsiourvas

We study the problem of learning optimal policy from a set of discrete treatment options using observational data. We propose a piecewise linear neural network model that can balance strong prescriptive performance and interpretability, which we refer to as the prescriptive ReLU network, or P-ReLU. We show analytically that this model (i) partitions the input space into disjoint polyhedra, where all instances that belong to the same partition receive the same treatment, and (ii) can be converted into an equivalent prescriptive tree with hyperplane splits for interpretability. We demonstrate the flexibility of the P-ReLU network as constraints can be easily incorporated with minor modifications to the architecture. Through experiments, we validate the superior prescriptive accuracy of P-ReLU against competing benchmarks. Lastly, we present examples of prescriptive trees extracted from trained P-ReLUs using a real-world dataset, for both the unconstrained and constrained scenarios.

Details

IJCAI Conference 2023 Conference Paper

MM-PCQA: Multi-Modal Learning for No-reference Point Cloud Quality Assessment

  • Zicheng Zhang
  • Wei Sun
  • Xiongkuo Min
  • Qiyuan Wang
  • Jun He
  • Quan Zhou
  • Guangtao Zhai

The visual quality of point clouds has been greatly emphasized since the ever-increasing 3D vision applications are expected to provide cost-effective and high-quality experiences for users. Looking back on the development of point cloud quality assessment (PCQA), the visual quality is usually evaluated by utilizing single-modal information, i. e. , either extracted from the 2D projections or 3D point cloud. The 2D projections contain rich texture and semantic information but are highly dependent on viewpoints, while the 3D point clouds are more sensitive to geometry distortions and invariant to viewpoints. Therefore, to leverage the advantages of both point cloud and projected image modalities, we propose a novel no-reference Multi-Modal Point Cloud Quality Assessment (MM-PCQA) metric. In specific, we split the point clouds into sub-models to represent local geometry distortions such as point shift and down-sampling. Then we render the point clouds into 2D image projections for texture feature extraction. To achieve the goals, the sub-models and projected images are encoded with point-based and image-based neural networks. Finally, symmetric cross-modal attention is employed to fuse multi-modal quality-aware information. Experimental results show that our approach outperforms all compared state-of-the-art methods and is far ahead of previous no-reference PCQA methods, which highlights the effectiveness of the proposed method. The code is available at https: //github. com/zzc-1998/MM-PCQA.

PDF Details DOI

AAAI Conference 2023 Conference Paper

Scalable Optimal Multiway-Split Decision Trees with Constraints

  • Shivaram Subramanian
  • Wei Sun

There has been a surge of interest in learning optimal decision trees using mixed-integer programs (MIP) in recent years, as heuristic-based methods do not guarantee optimality and find it challenging to incorporate constraints that are critical for many practical applications. However, existing MIP methods that build on an arc-based formulation do not scale well as the number of binary variables is in the order of 2 to the power of the depth of the tree and the size of the dataset. Moreover, they can only handle sample-level constraints and linear metrics. In this paper, we propose a novel path-based MIP formulation where the number of decision variables is independent of dataset size. We present a scalable column generation framework to solve the MIP. Our framework produces a multiway-split tree which is more interpretable than the typical binary-split trees due to its shorter rules. Our framework is more general as it can handle nonlinear metrics such as F1 score, and incorporate a broader class of constraints. We demonstrate its efficacy with extensive experiments. We present results on datasets containing up to 1,008,372 samples while existing MIP-based decision tree models do not scale well on data beyond a few thousand points. We report superior or competitive results compared to the state-of-art MIP-based methods with up to a 24X reduction in runtime.

PDF Details DOI

AAAI Conference 2022 Conference Paper

Constrained Prescriptive Trees via Column Generation

  • Shivaram Subramanian
  • Wei Sun
  • Youssef Drissi
  • Markus Ettl

With the abundance of available data, many enterprises seek to implement data-driven prescriptive analytics to help them make informed decisions. These prescriptive policies need to satisfy operational constraints, and proactively eliminate rule conflicts, both of which are ubiquitous in practice. It is also desirable for them to be simple and interpretable, so they can be easily verified and implemented. Existing approaches from the literature center around constructing variants of prescriptive decision trees to generate interpretable policies. However, none of the existing methods are able to handle constraints. In this paper, we propose a scalable method that solves the constrained prescriptive policy generation problem. We introduce a novel path-based mixed-integer program (MIP) formulation which identifies a (near) optimal policy efficiently via column generation. The policy generated can be represented as a multiway-split tree which is more interpretable and informative than a binary-split tree due to its shorter rules. We demonstrate the efficacy of our method with extensive experiments on both synthetic and real datasets.

PDF Details

AAAI Conference 2022 Conference Paper

Enhancing Counterfactual Classification Performance via Self-Training

  • Ruijiang Gao
  • Max Biggs
  • Wei Sun
  • Ligong Han

Unlike traditional supervised learning, in many settings only partial feedback is available. We may only observe outcomes for the chosen actions, but not the counterfactual outcomes associated with other alternatives. Such settings encompass a wide variety of applications including pricing, online marketing and precision medicine. A key challenge is that observational data are influenced by historical policies deployed in the system, yielding a biased data distribution. We approach this task as a domain adaptation problem and propose a selftraining algorithm which imputes outcomes with categorical values for finite unseen actions in the observational data to simulate a randomized trial through pseudolabeling, which we refer to as Counterfactual Self-Training (CST). CST iteratively imputes pseudolabels and retrains the model. In addition, we show input consistency loss can further improve CST performance which is shown in recent theoretical analysis of pseudolabeling. We demonstrate the effectiveness of the proposed algorithms on both synthetic and real datasets.

PDF Details

AAAI Conference 2022 Conference Paper

HNO: High-Order Numerical Architecture for ODE-Inspired Deep Unfolding Networks

  • Lin Kong
  • Wei Sun
  • Fanhua Shang
  • Yuanyuan Liu
  • Hongying Liu

Recently, deep unfolding networks (DUNs) based on optimization algorithms have received increasing attention, and their high efficiency has been confirmed by many experimental and theoretical results. Since this type of networks combines model-based traditional optimization algorithms, they have high interpretability. In addition, ordinary differential equations (ODEs) are often used to explain deep neural networks, and provide some inspiration for designing innovative network models. In this paper, we transform DUNs into first-order ODE forms, and propose a high-order numerical architecture for ODE-inspired deep unfolding networks. To the best of our knowledge, this is the first work to establish the relationship between DUNs and ODEs. Moreover, we take two representative DUNs as examples, apply our architecture to them and design novel DUNs. In theory, we prove the existence, uniqueness of the solution and convergence of the proposed network, and also prove that our network obtains a fast linear convergence rate. Extensive experiments verify the effectiveness and advantages of our architecture.

PDF Details

TIST Journal 2022 Journal Article

Multitask Balanced and Recalibrated Network for Medical Code Prediction

  • Wei Sun
  • Shaoxiong Ji
  • Erik Cambria
  • Pekka Marttinen

Human coders assign standardized medical codes to clinical documents generated during patients’ hospitalization, which is error prone and labor intensive. Automated medical coding approaches have been developed using machine learning methods, such as deep neural networks. Nevertheless, automated medical coding is still challenging because of complex code association, noise in lengthy documents, and the imbalanced class problem. We propose a novel neural network, called the Multitask Balanced and Recalibrated Neural Network, to solve these issues. Significantly, the multitask learning scheme shares the relationship knowledge between different coding branches to capture code association. A recalibrated aggregation module is developed by cascading convolutional blocks to extract high-level semantic features that mitigate the impact of noise in documents. Also, the cascaded structure of the recalibrated module can benefit learning from lengthy notes. To solve the imbalanced class problem, we deploy focal loss to redistribute the attention on low- and high-frequency medical codes. Experimental results show that our proposed model outperforms competitive baselines on a real-world clinical dataset called the Medical Information Mart for Intensive Care (MIMIC-III).

Details DOI

NeurIPS Conference 2022 Conference Paper

Video-based Human-Object Interaction Detection from Tubelet Tokens

  • Danyang Tu
  • Wei Sun
  • Xiongkuo Min
  • Guangtao Zhai
  • Wei Shen

We present a novel vision Transformer, named TUTOR, which is able to learn tubelet tokens, served as highly-abstracted spatial-temporal representations, for video-based human-object interaction (V-HOI) detection. The tubelet tokens structurize videos by agglomerating and linking semantically-related patch tokens along spatial and temporal domains, which enjoy two benefits: 1) Compactness: each token is learned by a selective attention mechanism to reduce redundant dependencies from others; 2) Expressiveness: each token is enabled to align with a semantic instance, i. e. , an object or a human, thanks to agglomeration and linking. The effectiveness and efficiency of TUTOR are verified by extensive experiments. Results show our method outperforms existing works by large margins, with a relative mAP gain of $16. 14\%$ on VidHOI and a 2 points gain on CAD-120 as well as a $4 \times$ speedup.

PDF Details

NeurIPS Conference 2021 Conference Paper

DominoSearch: Find layer-wise fine-grained N:M sparse schemes from dense neural networks

  • Wei Sun
  • Aojun Zhou
  • Sander Stuijk
  • Rob Wijnhoven
  • Andrew O. Nelson
  • Hongsheng Li
  • Henk Corporaal

Neural pruning is a widely-used compression technique for Deep Neural Networks (DNNs). Recent innovations in Hardware Architectures (e. g. Nvidia Ampere Sparse Tensor Core) and N: M fine-grained Sparse Neural Network algorithms (i. e. every M-weights contains N non-zero values) reveal a promising research line of neural pruning. However, the existing N: M algorithms only address the challenge of how to train N: M sparse neural networks in a uniform fashion (i. e. every layer has the same N: M sparsity) and suffer from a significant accuracy drop for high sparsity (i. e. when sparsity > 80\%). To tackle this problem, we present a novel technique -- \textbf{\textit{DominoSearch}} to find mixed N: M sparsity schemes from pre-trained dense deep neural networks to achieve higher accuracy than the uniform-sparsity scheme with equivalent complexity constraints (e. g. model size or FLOPs). For instance, for the same model size with 2. 1M parameters (87. 5\% sparsity), our layer-wise N: M sparse ResNet18 outperforms its uniform counterpart by 2. 1\% top-1 accuracy, on the large-scale ImageNet dataset. For the same computational complexity of 227M FLOPs, our layer-wise sparse ResNet18 outperforms the uniform one by 1. 3\% top-1 accuracy. Furthermore, our layer-wise fine-grained N: M sparse ResNet50 achieves 76. 7\% top-1 accuracy with 5. 0M parameters. {This is competitive to the results achieved by layer-wise unstructured sparsity} that is believed to be the upper-bound of Neural Network pruning with respect to the accuracy-sparsity trade-off. We believe that our work can build a strong baseline for further sparse DNN research and encourage future hardware-algorithm co-design work. Our code and models are publicly available at \url{https: //github. com/NM-sparsity/DominoSearch}.

PDF Details

YNIMG Journal 2020 Journal Article

A probabilistic atlas of locus coeruleus pathways to transentorhinal cortex for connectome imaging in Alzheimer's disease

  • Wei Sun
  • Yuchun Tang
  • Yuchuan Qiao
  • Xinting Ge
  • Mara Mather
  • John M. Ringman
  • Yonggang Shi

According to the latest Braak staging of Alzheimer's disease (AD), tau pathology occurs earliest in the brain in the locus coeruleus (LC) of the brainstem, then propagates to the transentorhinal cortex (TEC), and later to other neocortical regions. Recent animal and in vivo human brain imaging research also support the trans-axonal propagation of tau pathology. In addition, neurochemical studies link norepinephrine to behavioral symptoms in AD. It is thus critical to examine the integrity of the LC-TEC pathway in studying the early development of the disease, but there has been limited work in this direction. By leveraging the high-resolution and multi-shell diffusion MRI data from the Human Connectome Project (HCP), in this work we develop a novel method for the reconstruction of the LC-TEC pathway in a cohort of 40 HCP subjects carefully selected based on rigorous quality control of the residual distortion artifacts in the brainstem. A probabilistic atlas of the LC-TEC pathway of both hemispheres is then developed in the MNI152 space and distributed publicly on the NITRC website. To apply our atlas on clinical imaging data, we develop an automated approach to calculate the medial core of the LC-TEC pathway for localized analysis of connectivity changes. In a cohort of 138 subjects from the Alzheimer's Disease Neuroimaging Initiative (ADNI), we demonstrate the detection of the decreased fiber integrity in the LC-TEC pathways with increasing disease severity.

Details DOI

AAAI Conference 2020 Conference Paper

Fatigue-Aware Bandits for Dependent Click Models

  • Junyu Cao
  • Wei Sun
  • Zuo-Jun (Max) Shen
  • Markus Ettl

As recommender systems send a massive amount of content to keep users engaged, users may experience fatigue which is contributed by 1) an overexposure to irrelevant content, 2) boredom from seeing too many similar recommendations. To address this problem, we consider an online learning setting where a platform learns a policy to recommend content that takes user fatigue into account. We propose an extension of the Dependent Click Model (DCM) to describe users’ behavior. We stipulate that for each piece of content, its attractiveness to a user depends on its intrinsic relevance and a discount factor which measures how many similar contents have been shown. Users view the recommended content sequentially and click on the ones that they find attractive. Users may leave the platform at any time, and the probability of exiting is higher when they do not like the content. Based on user’s feedback, the platform learns the relevance of the underlying content as well as the discounting effect due to content fatigue. We refer to this learning task as “fatigue-aware DCM Bandit” problem. We consider two learning scenarios depending on whether the discounting effect is known. For each scenario, we propose a learning algorithm which simultaneously explores and exploits, and characterize its regret bound.

PDF Details

AAAI Conference 2019 Conference Paper

Dynamic Learning of Sequential Choice Bandit Problem under Marketing Fatigue

  • Junyu Cao
  • Wei Sun

Motivated by the observation that overexposure to unwanted marketing activities leads to customer dissatisfaction, we consider a setting where a platform offers a sequence of messages to its users and is penalized when users abandon the platform due to marketing fatigue. We propose a novel sequential choice model to capture multiple interactions taking place between the platform and its user: Upon receiving a message, a user decides on one of the three actions: accept the message, skip and receive the next message, or abandon the platform. Based on user feedback, the platform dynamically learns users’ abandonment distribution and their valuations of messages to determine the length of the sequence and the order of the messages, while maximizing the cumulative payoff over a horizon of length T. We refer to this online learning task as the sequential choice bandit problem. For the offline combinatorial optimization problem, we show a polynomialtime algorithm. For the online problem, we propose an algorithm that balances exploration and exploitation, and characterize its regret bound. Lastly, we demonstrate how to extend the model with user contexts to incorporate personalization.

PDF Details

YNIMG Journal 2019 Journal Article

FOD-based registration for susceptibility distortion correction in brainstem connectome imaging

  • Yuchuan Qiao
  • Wei Sun
  • Yonggang Shi

The high resolution, multi-shell diffusion MRI (dMRI) data from the Human Connectome Project (HCP) provides a great opportunity to map fine-grained fiber pathways in human brainstem, but the severe susceptibility-induced distortion around the brainstem poses a significant challenge. While the correction tools used in the HCP Pipeline greatly reduce the distortion artifacts in the preprocessed data, significant residual distortions are still widely present, especially in the brainstem region. One fundamental reason is that the topup tool used in the HCP Pipeline only relies on the B0 images, which lack sufficient contrast about white matter pathways, to estimate the distortion displacement between opposite phase encodings (PEs). To fully utilize the rich information of HCP data that includes dMRI data from two opposite PEs, we compute the fiber orientation distributions (FODs) from the data of each PE and propose a novel method to estimate and correct the residual distortion using FOD-based registration. Using the dMRI data of 94 HCP subjects, we show quantitatively that our method can reduce the misalignment of main fiber direction in the brainstem by 21% as compared to the topup tool used in the HCP Pipeline. Our method is fully compatible with the HCP Pipeline and thus can be readily integrated with it to enhance distortion correction in connectome imaging research.

Details DOI

IJCAI Conference 2019 Conference Paper

Learn to Select via Hierarchical Gate Mechanism for Aspect-Based Sentiment Analysis

  • Xiangying Ran
  • Yuanyuan Pan
  • Wei Sun
  • Chongjun Wang

Aspect-based sentiment analysis (ABSA) is a fine-grained task. Recurrent Neural Network (RNN) model armed with attention mechanism seems a natural fit for this task, and actually it achieves the state-of-the-art performance recently. However, previous attention mechanisms proposed for ABSA may attend irrelevant words and thus downgrade the performance, especially when dealing with long and complex sentences with multiple aspects. In this paper, we propose a novel architecture named Hierarchical Gate Memory Network (HGMN) for ABSA: firstly, we employ the proposed hierarchical gate mechanism to learn to select the related part about the given aspect, which can keep the original sequence structure of sentence at the same time. After that, we apply Convolutional Neural Network (CNN) on the final aspect-specific memory. We conduct extensive experiments on the SemEval 2014 and Twitter dataset, and results demonstrate that our model outperforms attention based state-of-the-art baselines.

PDF Details

JBHI Journal 2019 Journal Article

Weakly Supervised Biomedical Image Segmentation by Reiterative Learning

  • Qiaokang Liang
  • Yang Nan
  • Gianmarc Coppola
  • Kunglin Zou
  • Wei Sun
  • Dan Zhang
  • Yaonan Wang
  • Guanzhen Yu

Recent advances in deep learning have produced encouraging results for biomedical image segmentation; however, outcomes rely heavily on comprehensive annotation. In this paper, we propose a neural network architecture and a new algorithm, known as overlapped region forecast, for the automatic segmentation of gastric cancer images. To the best of our knowledge, this report for the first time describes that deep learning has been applied to the segmentation of gastric cancer images. Moreover, a reiterative learning framework that achieves superior performance without pretraining or further manual annotation is presented to train a simple network on weakly annotated biomedical images. We customize the loss function to make the model converge faster while avoiding becoming trapped in local minima. Patch boundary errors were eliminated by our overlapped region forecast algorithm. By studying the characteristics of the model trained using two different patch extraction methods, we train iteratively and integrate predictions and weak annotations to improve the quality of the training data. Using these methods, a mean Intersection over Union coefficient of 0. 883 and a mean accuracy of 91. 09% were achieved on the partially labeled dataset, thereby securing a win in the 2017 China Big Data and Artificial Intelligence Innovation and Entrepreneurship Competition.

Details DOI

YNIMG Journal 2018 Journal Article

A probabilistic atlas of human brainstem pathways based on connectome imaging data

  • Yuchun Tang
  • Wei Sun
  • Arthur W. Toga
  • John M. Ringman
  • Yonggang Shi

The brainstem is a critical structure that regulates vital autonomic functions, houses the cranial nerves and their nuclei, relays motor and sensory information between the brain and spinal cord, and modulates cognition, mood, and emotions. As a primary relay center, the fiber pathways of the brainstem include efferent and afferent connections among the cerebral cortex, spinal cord, and cerebellum. While diffusion MRI has been successfully applied to map various brain pathways, its application for the in vivo imaging of the brainstem pathways has been limited due to inadequate resolution and large susceptibility-induced distortion artifacts. With the release of high-resolution data from the Human Connectome Project (HCP), there is increasing interest in mapping human brainstem pathways. Previous works relying on HCP data to study brainstem pathways, however, did not consider the prevalence (>80%) of large distortions in the brainstem even after the application of correction procedures from the HCP-Pipeline. They were also limited in the lack of adequate consideration of subject variability in either fiber pathways or region of interests (ROIs) used for bundle reconstruction. To overcome these limitations, we develop in this work a probabilistic atlas of 23 major brainstem bundles using high-quality HCP data passing rigorous quality control. For the large-scale data from the 500-Subject release of HCP, we conducted extensive quality controls to exclude subjects with severe distortions in the brainstem area. After that, we developed a systematic protocol to manually delineate 1300 ROIs on 20 HCP subjects (10 males; 10 females) for the reconstruction of fiber bundles using tractography techniques. Finally, we leveraged our novel connectome modeling techniques including high order fiber orientation distribution (FOD) reconstruction from multi-shell diffusion imaging and topography-preserving tract filtering algorithms to successfully reconstruct the 23 fiber bundles for each subject, which were then used to calculate the probabilistic atlases in the MNI152 space for public release. In our experimental results, we demonstrate that our method yielded anatomically faithful reconstruction of the brainstem pathways and achieved improved performance in comparison with an existing atlas of cerebellar peduncles based on HCP data. These atlases have been publicly released on NITRIC (https: //www. nitrc. org/projects/brainstem_atlas/) and can be readily used by brain imaging researchers interested in studying brainstem pathways.

Details DOI

JAIR Journal 2018 Journal Article

Graphical Model Market Maker for Combinatorial Prediction Markets

  • Kathryn Blackmond Laskey
  • Wei Sun
  • Robin Hanson
  • Charles Twardy
  • Shou Matsumoto
  • Brandon Goldfedder

We describe algorithms for use by prediction markets in forming a crowd consensus joint probability distribution over thousands of related events. Equivalently, we describe market mechanisms to efficiently crowdsource both structure and parameters of a Bayesian network. Prediction markets are among the most accurate methods to combine forecasts; forecasters form a consensus probability distribution by trading contingent securities. A combinatorial prediction market forms a consensus joint distribution over many related events by allowing conditional trades or trades on Boolean combinations of events. Explicitly representing the joint distribution is infeasible, but standard inference algorithms for graphical probability models render it tractable for large numbers of base events. We show how to adapt these algorithms to compute expected assets conditional on a prospective trade, and to find the conditional state where a trader has minimum assets, allowing full asset reuse. We compare the performance of three algorithms: the straightforward algorithm from the DAGGRE (Decomposition-Based Aggregation) prediction market for geopolitical events, the simple block-merge model from the SciCast market for science and technology forecasting, and a more sophisticated algorithm we developed for future markets.

PDF Details DOI

NeurIPS Conference 2018 Conference Paper

Sketching Method for Large Scale Combinatorial Inference

  • Wei Sun
  • Junwei Lu
  • Han Liu

We present computationally efficient algorithms to test various combinatorial structures of large-scale graphical models. In order to test the hypotheses on their topological structures, we propose two adjacency matrix sketching frameworks: neighborhood sketching and subgraph sketching. The neighborhood sketching algorithm is proposed to test the connectivity of graphical models. This algorithm randomly subsamples vertices and conducts neighborhood regression and screening. The global sketching algorithm is proposed to test the topological properties requiring exponential computation complexity, especially testing the chromatic number and the maximum clique. This algorithm infers the corresponding property based on the sampled subgraph. Our algorithms are shown to substantially accelerate the computation of existing methods. We validate our theory and method through both synthetic simulations and a real application in neuroscience.

PDF Details

AAAI Conference 2015 Conference Paper

Causal Inference via Sparse Additive Models with Application to Online Advertising

  • Wei Sun
  • Pengyuan Wang
  • Dawei Yin
  • Jian Yang
  • Yi Chang

Advertising effectiveness measurement is a fundamental problem in online advertising. Various causal inference methods have been employed to measure the causal effects of ad treatments. However, existing methods mainly focus on linear logistic regression for univariate and binary treatments and are not well suited for complex ad treatments of multi-dimensions, where each dimension could be discrete or continuous. In this paper we propose a novel two-stage causal inference framework for assessing the impact of complex ad treatments. In the first stage, we estimate the propensity parameter via a sparse additive model; in the second stage, a propensity-adjusted regression model is applied for measuring the treatment effect. Our approach is shown to provide an unbiased estimation of the ad effectiveness under regularity conditions. To demonstrate the efficacy of our approach, we apply it to a real online advertising campaign to evaluate the impact of three ad treatments: ad frequency, ad channel, and ad size. We show that the ad frequency usually has a treatment effect cap when ads are showing on mobile device. In addition, the strategies for choosing best ad size are completely different for mobile ads and online ads.

PDF Details

NeurIPS Conference 2015 Conference Paper

Non-convex Statistical Optimization for Sparse Tensor Graphical Model

  • Wei Sun
  • Zhaoran Wang
  • Han Liu
  • Guang Cheng

We consider the estimation of sparse graphical models that characterize the dependency structure of high-dimensional tensor-valued data. To facilitate the estimation of the precision matrix corresponding to each way of the tensor, we assume the data follow a tensor normal distribution whose covariance has a Kronecker product structure. The penalized maximum likelihood estimation of this model involves minimizing a non-convex objective function. In spite of the non-convexity of this estimation problem, we prove that an alternating minimization algorithm, which iteratively estimates each sparse precision matrix while fixing the others, attains an estimator with the optimal statistical rate of convergence as well as consistent graph recovery. Notably, such an estimator achieves estimation consistency with only one tensor sample, which is unobserved in previous work. Our theoretical results are backed by thorough numerical studies.

PDF Details

EAAI Journal 2014 Journal Article

Multi-objective ecological reservoir operation based on water quality response models and improved genetic algorithm: A case study in Three Gorges Reservoir, China

  • Ming Hu
  • Guo H. Huang
  • Wei Sun
  • Yongping Li
  • Xiaowen Ding
  • Chunjiang An
  • Xiaofei Zhang
  • Ting Li

This study proposes a self-adaptive GA-aided multi-objective ecological reservoir operation model (SMEROM) and applies it to water quality management in the Xiangxi River near to the Three Gorges Reservoir, China. The SMEROM integrates statistical water quality models, multi-objective reservoir operations, and a self-adaptive GA within a general framework. Among them, the statistical water quality models of the Xiangxi River are formulated to deal with the relationships between reservoir operation and water quality, which are embedded in constraints of the SMEROM. The multiple objective functions, including maximizing hydropower generation, minimizing loss of flood control, minimizing rate of flood risk, maximizing the average remaining capacity of flood control and maximizing the benefit of shipping, are considered simultaneously to obtain comprehensive benefit among the environment, society and economy. The weighting method is employed to convert the multiple objectives to a single objective. To solve the complex SMEROM, an improved self-adaptive GA is employed through incorporating simulated binary crossover and self-adaptive mutation. To demonstrate the advantage of the developed SMEROM model, the solutions through ecological reservoir operation are compared with those through the traditional reservoir operation and the practical operation in 2011, in terms of water quality, reservoir operation and objective function values. The results show that most of benefit in the ecological operation is better than that in the traditional or practical operations except for the hydropower benefit and loss benefit of flood control. This is because flood control and environmental protection are reasonably considered in the ecological operation.

Details DOI

JMLR Journal 2013 Journal Article

Consistent Selection of Tuning Parameters via Variable Selection Stability

  • Wei Sun
  • Junhui Wang
  • Yixin Fang

Penalized regression models are popularly used in high- dimensional data analysis to conduct variable selection and model fitting simultaneously. Whereas success has been widely reported in literature, their performances largely depend on the tuning parameters that balance the trade-off between model fitting and model sparsity. Existing tuning criteria mainly follow the route of minimizing the estimated prediction error or maximizing the posterior model probability, such as cross validation, AIC and BIC. This article introduces a general tuning parameter selection criterion based on variable selection stability. The key idea is to select the tuning parameters so that the resultant penalized regression model is stable in variable selection. The asymptotic selection consistency is established for both fixed and diverging dimensions. Its effectiveness is also demonstrated in a variety of simulated examples as well as an application to the prostate cancer data. [abs] [ pdf ][ bib ] &copy JMLR 2013. ( edit, beta )

PDF Details

EAAI Journal 2011 Journal Article

Revealing digital fakery using multiresolution decomposition and higher order statistics

  • Wei Lu
  • Wei Sun
  • Fu-lai Chung
  • Hongtao Lu

With the advance of digitization and digital processing techniques, digital images are now easy to create and manipulate, and leave no clues of artificial evidence. There are some known digital fakery for images, e. g. , computer graphics (CGs) and digital forgeries. As valid records of natural world, natural images, i. e. , photographic images, are no longer believable. In this paper, a detection scheme for natural images and fake images is proposed. Features are first extracted using multiresolution decomposition and higher order local autocorrelations (HLACs). The support vector machines (SVMs) are then used to differentiate the natural and fake images. Because the inner product between features can be obtained directly without computing features, it can be integrated into SVM, and the computation complexity is decreased. Experiments show that the proposed detection scheme is effective, demonstrating that the proposed statistical features can model the differences between natural images and fake images.

Details DOI
v2026.09.13