Wei Huang

EAAI Journal 2026 Journal Article

A multi-channel adaptive neural network for querying the optimal time-varying damage route with collective spatial keywords

• Zhilei Xu
• Wei Huang

In geographic information system services, the optimal route planning with collective spatial keywords plays a crucial role in providing efficient and feasible travel solutions. Existing research on damage conditions of points of interest in road networks over time remains incomplete. To address this issue, we propose an innovative multi-channel adaptive neural network model and algorithm for querying the optimal path with collective spatial keywords on a time-varying damage network. To address the variations in edge lengths in time-varying networks, we have designed multi-channel gated neurons that incorporate node state judgment, departure time selection, and transmission time control. These neurons are integrated with a logic gate to manage the time-varying edge lengths. To ensure the accuracy of data exchange, we have introduced a multi-channel mechanism for data isolation. We have analyzed the time complexity and correctness of the proposed algorithms and conducted comparative experiments using a public road network dataset. The experimental results demonstrate the effectiveness and superiority of the proposed method in solving the optimal path with collective spatial keywords query problem on time-varying damage networks, providing technical support for intelligent traffic planning projects.

JBHI Journal 2026 Journal Article

A Tri-Factor Adaptive Federated Learning Framework for Parkinson’s Disease Diagnosis via Multi-Source Facial Expression Analysis

• Meng Pang
• Houwei Xu
• Zheng Huang
• Yintao Zhou
• Shengbo Chen
• Binghui Wang
• Wei Huang

Early diagnosis of Parkinson’s disease (PD) is crucial for timely treatment and disease management. Recent studies link PD to impaired facial muscle control, manifesting as “masked face” symptoms, offering a novel diagnostic approach through facial expression analysis. However, data privacy concerns and legal restrictions have resulted in significant “data silos”, hindering data sharing and limiting the accuracy and generalizability of existing diagnostic models due to small, localized datasets. To address these challenges, we propose an innovative Tri-Factor Adaptive Federated Learning (TriAFL) framework, designed to collaboratively analyze facial expression data across multiple medical institutions while ensuring robust data privacy protection. TriAFL introduces a comprehensive evaluation mechanism that assesses client contributions across three dimensions: gradient, data, and learning efficiency, effectively addressing Non-IID issues arising from data size variations and heterogeneity. To validate the real-world applicability of our method, we collaborate with a hospital to build the largest known facial expression dataset of PD patients. Furthermore, we explore the integration of local data augmentation strategy to further enhance diagnostic accuracy. Comprehensive experimental results demonstrate TriAFL’s superior performance over conventional FL methods in classification task, as well as confirms TriAFL’s efficacy in PD diagnosis, delivering a rapid, non-invasive screening tool while driving advancements in AI-powered healthcare.

AAAI Conference 2026 Conference Paper

CoEvoer: Collaborative Evolution Transformer for Upper-Body Expressive Human Pose and Shape Estimation

• Yuxiang Zhao
• Wei Huang
• Yujie Song
• Liu Wang
• Huan Zhao

Expressive Human Pose and Shape Estimation (EHPS) plays a crucial role in various AR/VR applications and has witnessed significant progress in recent years. However, current state-of-the-art methods still struggle with accurate parameter estimation for facial and hand regions and exhibit limited generalization to wild images. To address these challenges, we present CoEvoer, a novel one-stage synergistic cross-dependency transformer framework tailored for upper-body EHPS. CoEvoer enables explicit feature-level interaction across different body parts, allowing for mutual enhancement through contextual information exchange. Specifically, larger and more easily estimated regions such as the torso provide global semantics and positional priors to guide the estimation of finer, more complex regions like the face and hands. Conversely, the localized details captured in facial and hand regions help refine and calibrate adjacent body parts. To the best of our knowledge, CoEvoer is the first framework designed specifically for upper-body EHPS, with the goal of capturing the strong coupling and semantic dependencies among the face, hands, and torso through joint parameter regression. Extensive experiments demonstrate that CoEvoer achieves state-of-the-art performance on upper-body benchmarks and exhibits strong generalization capability even on unseen wild images.

JBHI Journal 2026 Journal Article

Contactless Hemodynamic Monitoring Based on Multi-Scale Gaussian Filtering via Imaging PPG

• Yonggang Tong
• Zhipei Huang
• Feng Qiu
• Chenhao Wu
• Xiaoyong Tao
• Wei Huang
• Fei Qin

Imaging photoplethysmography (iPPG) is an emerging optical technique that allows for the contactless acquisition of arterial Blood Volume Pulse (BVP) signals from video recordings of the human skin. While iPPG offers a non-contact and convenient means for physiological monitoring, the accuracy of the extracted BVP signals remains limited. This limitation hinders its potential for advanced cardiovascular assessments, such as evaluations of arterial stiffness and cardiac function. To address this issue, we propose a novel physiologically informed Gaussian filtering method, based on the prior knowledge that the BVP waveform can be modeled as a mixture of multiple Gaussian components. Specifically, a set of physiological Gaussian kernels is employed to convolve the noisy iPPG signal, generating a Gaussian representation that emphasizes waveform components with physiological relevance. This representation is further refined by a Transformer-based neural network to reconstruct accurate BVP signals. Experimental results demonstrate a notable improvement in BVP accuracy, with the mean absolute error reducing from 0. 25 to 0. 08. This enhancement in iPPG precision highlights the potential of our approach for advanced medical applications.

AAAI Conference 2026 Conference Paper

DualFete: Revisiting Teacher-Student Interactions from a Feedback Perspective for Semi-supervised Medical Image Segmentation

• Le Yi
• Wei Huang
• Lei Zhang
• Kefu Zhao
• Yan Wang
• Zizhou Wang

The teacher-student paradigm has emerged as a canonical framework in semi-supervised learning. When applied to medical image segmentation, the paradigm faces challenges due to inherent image ambiguities, making it particularly vulnerable to erroneous supervision. Crucially, the student's iterative reconfirmation of these errors leads to self-reinforcing bias. While some studies attempt to mitigate this bias, they often rely on external modifications to the conventional teacher-student framework, overlooking its intrinsic potential for error correction. In response, this work introduces a feedback mechanism into the teacher-student framework to counteract error reconfirmations. Here, the student provides feedback on the changes induced by the teacher's pseudo-labels, enabling the teacher to refine these labels accordingly. We specify that this interaction hinges on two key components: the feedback attributor, which designates pseudo-labels triggering the student's update, and the feedback receiver, which determines where to apply this feedback. Building on this, a dual-teacher feedback model is further proposed, which allows more dynamics in the feedback loop and fosters more gains by resolving disagreements through cross-teacher supervision while avoiding consistent errors. Comprehensive evaluations on three medical image benchmarks demonstrate the method's effectiveness in addressing error propagation in semi-supervised medical image segmentation.

JBHI Journal 2026 Journal Article

JS-RegNeXt: A ConvNeXt-based few-shot JSR framework with correlation awareness and multi-scale prediction consistency

• Teng Li
• Runing Xiao
• Tongtong Xie
• Wei Huang
• Jialong Hou
• Yuchuan Qiao
• Changyan Xiao

Conventional label-constrained (LC) medical image registration methods are extremely dependent on the number of labels, resulting in the overfitting problem when the number of labels is insufficient. Recently, joint segmentation and registration (JSR) methods have demonstrated promising results for the LC registration tasks in few-shot situations. However, these methods typically lack global correlation awareness of the images to be registered and cannot robustly perceive global semantic information, leading to suboptimal registration performance on anatomy with low contrast or blurred boundaries. Therefore, we propose a novel JS-RegNeXt framework for few-shot label-constrained registration for medical images, which consists of segmentation and registration modules. Specifically, the segmentation module perceives global semantic information, and the registration module generates synthetic labeled data to fine-tune it. For the segmentation module, a SegNet with multi-scale prediction consistency is designed to mitigate the uncertainty introduced by synthesized data and to improve the robustness of semantic perception, even in low-contrast regions. For the registration module, a RegNeXt is proposed to achieve correlation awareness between images and leverages the large receptive field of ConvNeXt to enhance global perception. This design improves robustness in low-contrast regions, leading to more accurate and reliable image registration. Experiments on two public 3D medical image datasets, cardiac CT and brain MRI, show that our JS-RegNeXt achieves improvements in both segmentation and registration tasks compared to many state-of-the-art methods. It demonstrates that our JS-RegNeXt framework has great potential for clinical application.

AAAI Conference 2026 Conference Paper

On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGD

• Tongcheng Zhang
• Zhanpeng Zhou
• Mingze Wang
• Andi Han
• Wei Huang
• Taiji Suzuki
• Junchi Yan

One crucial factor behind the success of deep learning lies in the implicit bias induced by noise inherent in gradient-based training algorithms. Motivated by empirical observations that training with noisy labels improves model generalization, we delve into the underlying mechanisms behind stochastic gradient descent (SGD) with label noise. Focusing on a two-layer over-parameterized linear network, we analyze the learning dynamics of label noise SGD, unveiling a two-phase learning behavior. In Phase I, the magnitudes of model weights progressively diminish, and the model escapes the lazy regime; enters the rich regime. In Phase II, the alignment between model weights and the ground-truth interpolator increases, and the model eventually converges. Our analysis highlights the critical role of label noise in driving the transition from the lazy to the rich regime and minimally explains its empirical success. Furthermore, we extend these insights to Sharpness-Aware Minimization (SAM), showing that the principles governing label noise SGD also apply to broader optimization algorithms. Extensive experiments, conducted under both synthetic and real-world setups, strongly support our theory.

AAAI Conference 2026 Conference Paper

Prior Refinement Is Better: Diffusion-Driven Graph Harmonization for Federated Graph Learning

• Shuman Zhuang
• Zhihao Wu
• Wei Huang
• Luojun Lin
• Jia-Li Yin
• Lele Fu
• Hong-Ning Dai

Federated Graph Learning (FGL) has emerged as a compelling paradigm for collaboratively training a global model while preserving the privacy of multi-source graphs. Nonetheless, FGL faces a critical challenge of data heterogeneity, where semantic and structural discrepancies across clients significantly degrade its performance. Although existing methods attempt to calibrate client-specific graph distributions during federated training, they inevitably fall short in aligning the optimization behaviors across clients due to dynamic parameter updates, thereby inducing a bottleneck in generalization improvement. To tackle this challenge, we propose a solution from a new perspective of prior refinement, which seeks to proactively harmonize client graph distributions before the federated training. In particular, we propose a Federated Graph Harmonization (FedGH) framework that exploits the generative strengths of graph diffusion models to perform prior refinement of local graphs. In a nutshell, FedGH designs a conditional diffusion mechanism on each client that synthesizes pseudo-graphs encapsulating both feature and structural priors, thereby facilitating explicit correction of inter-client distributional bias. On the server side, we employ the graph contrastive learning between various client-specific pseudo-graphs to incorporate the global information, subsequently guiding local data reconstruction. Importantly, model-agnostic FedGH can be seamlessly deployed as a plug-and-play module to be easily integrated with existing FGL architectures. Extensive experiments demonstrate that FedGH consistently outperforms state-of-the-art FGL baselines.

AAAI Conference 2026 Conference Paper

SR-KI: Scalable and Real-Time Knowledge Integration into LLMs via Supervised Attention

• Bohan Yu
• Wei Huang
• Kang Liu

This paper proposes SR-KI, a novel approach for integrating real-time and large-scale structured knowledge bases (KBs) into large language models (LLMs). SR-KI begins by encoding KBs into key-value pairs using a pretrained encoder, and injects them into LLMs' KV cache. Building on this representation, we employ a two-stage training paradigm: first locating a dedicated retrieval layer within the LLM, and then applying an attention-based loss at this layer to explicitly supervise attention toward relevant KB entries. Unlike traditional retrieval-augmented generation methods that rely heavily on the performance of external retrievers and multi-stage pipelines, SR-KI supports end-to-end inference by performing retrieval entirely within the model’s latent space. This design enables efficient compression of injected knowledge and facilitates dynamic knowledge updates. Comprehensive experiments demonstrate that SR-KI enables the integration of up to 40K KBs into a 7B LLM on a single A100 40GB GPU, and achieves strong retrieval performance—maintaining over 98% Recall@10 on the best-performing task and exceeding 88% on average across all tasks. Task performance on question answering and KB ID generation also demonstrates that SR-KI maintains strong performance while achieving up to 99.75% compression of the injected KBs.

YNIMG Journal 2025 Journal Article

Accelerating multi-directional diffusion MRI through patch-based joint reconstruction

• Zhongbiao Xu
• Rongli Zhang
• Wei Huang
• Guanhua Deng
• Xiaoyun Liang
• Li Guo
• Junying Cheng
• Yaohui Wang

Diffusion magnetic resonance imaging (dMRI) is a valuable technique for studying tissue microstructure and connectivity in the brain. However, acquiring high-resolution dMRI data is time-consuming, limiting its clinical applicability. Traditional parallel imaging techniques can accelerate the acquisition of dMRI, but they are constrained by the geometry factor. In this study, we propose a novel patch-based multiple diffusion directions joint reconstruction method that simultaneously capitalizes on the intra- and inter-image correlation across multiple diffusion directions by grouping similar 3D image patches and then enforces the sparsity of these groups in sensitivity encoding (SENSE) reconstruction, termed PB-SENSE. The simulation and in vivo experiments demonstrated that the proposed method can achieve high-quality images comparable to those obtained from fully sampled data, even with an acceleration of 5. This suggests that the proposed method has the potential to enhance the practical application of high-resolution diffusion imaging.

AAAI Conference 2025 Conference Paper

Breaking Data Silos in Parkinson’s Disease Diagnosis: An Adaptive Federated Learning Approach for Privacy-Preserving Facial Expression Analysis

• Meng Pang
• Houwei Xu
• Zheng Huang
• Yintao Zhou
• Wei Huang
• Binghui Wang

The early diagnosis of Parkinson’s disease (PD) is crucial for potential patients to receive timely treatment and prevent disease progression. Recent studies have shown that PD is closely linked to impairments in facial muscle control, resulting in characteristic “masked face” symptoms. This discovery offers a novel perspective for PD diagnosis by leveraging facial expression recognition and analysis techniques to capture and quantify these features, thereby distinguishing between PD patients and non-PD individuals based on their facial expressions. However, concerns about data privacy and legal restrictions have led to significant “data silos”, posing challenges to data sharing and limiting the accuracy and generalization of existing diagnostic models due to small, localized datasets. To address this issue, we propose an innovative adaptive federated learning approach that aims to jointly analyze facial expression data from multiple medical institutions while preserving data privacy. Our proposed approach comprehensively evaluates each client's contributions in terms of gradient, data, and learning efficiency, overcoming the non-IID issues caused by varying data sizes or heterogeneity across clients. To demonstrate the real-world impact of our approach, we collected a new facial expression dataset of PD patients in collaboration with a hospital. Extensive experiments validate the effectiveness of our proposed method for PD diagnosis and facial expression recognition, offering a promising avenue for rapid, non-invasive initial screening and advancing healthcare intelligence.

ICML Conference 2025 Conference Paper

Can Diffusion Models Learn Hidden Inter-Feature Rules Behind Images?

• Yujin Han
• Andi Han
• Wei Huang
• Chaochao Lu
• Difan Zou

Despite the remarkable success of diffusion models (DMs) in data generation, they exhibit specific failure cases with unsatisfactory outputs. We focus on one such limitation: the ability of DMs to learn hidden rules between image features. Specifically, for image data with dependent features ($\mathbf{x}$) and ($\mathbf{y}$) (e. g. , the height of the sun ($\mathbf{x}$) and the length of the shadow ($\mathbf{y}$)), we investigate whether DMs can accurately capture the inter-feature rule ($p(\mathbf{y}|\mathbf{x})$). Empirical evaluations on mainstream DMs (e. g. , Stable Diffusion 3. 5) reveal consistent failures, such as inconsistent lighting-shadow relationships and mismatched object-mirror reflections. Inspired by these findings, we design four synthetic tasks with strongly correlated features to assess DMs’ rule-learning abilities. Extensive experiments show that while DMs can identify coarse-grained rules, they struggle with fine-grained ones. Our theoretical analysis demonstrates that DMs trained via denoising score matching (DSM) exhibit constant errors in learning hidden rules, as the DSM objective is not compatible with rule conformity. To mitigate this, we introduce a common technique - incorporating additional classifier guidance during sampling, which achieves (limited) improvements. Our analysis reveals that the subtle signals of fine-grained rules are challenging for the classifier to capture, providing insights for future exploration.

JMLR Journal 2025 Journal Article

Causal Effect of Functional Treatment

• Ruoxu Tan
• Wei Huang
• Zheng Zhang
• Guosheng Yin

We study the causal effect with a functional treatment variable, where practical applications often arise in neuroscience, biomedical sciences, etc. Previous research concerning the effect of a functional variable on an outcome is typically restricted to exploring correlation rather than causality. The generalized propensity score, which is often used to calibrate the selection bias, is not directly applicable to a functional treatment variable due to a lack of definition of probability density function for functional data. We propose three estimators for the average dose-response functional based on the functional linear model, namely, the functional stabilized weight estimator, the outcome regression estimator and the doubly robust estimator, each of which has its own merits. We study their theoretical properties, which are corroborated through extensive numerical experiments. A real data application on electroencephalography data and disease severity demonstrates the practical value of our methods. [abs] [ pdf ][ bib ] [ code ] &copy JMLR 2025. ( edit, beta )

EAAI Journal 2025 Journal Article

Efficient network architecture for target detection in challenging low-light environments

• Qiang Liu
• Wei Huang
• Tao Hu
• Xiaoqiu Duan
• Jie Yu
• Jiahuan Huang
• Jianghao Wei

Target detection in low-light and dark conditions presents significant challenges compared to normal scenarios due to blurriness, low contrast, and information loss from shadows and darkness. Color distortion and noise further hinder color-based detection algorithms. This study introduces an improved detection model that incorporates image enhancement techniques specifically designed for low-light conditions. The focus is on preprocessed image restoration to achieve realistic and natural visual quality. Key enhancements include a Multi-level Fine-Grained Feature Prediction Network architecture, enabling layered feature fusion and refined feature extraction. This leads to improved model accuracy, reduced parameters, and a dynamic detection output head that enhances target region precision, ultimately improving overall detection accuracy. Validation on a public dataset reveals a 5. 6% increase in mean average precision, a 6. 2% reduction in computational costs in terms of Floating-point Operations, and a 20% decrease in model parameter size. Our method outperforms mainstream low-light target detection algorithms in terms of performance and versatility.

ICLR Conference 2025 Conference Paper

From Layers to States: A State Space Model Perspective to Deep Neural Network Layer Dynamics

• Qinshuo Liu
• Weiqin Zhao
• Wei Huang
• Yanwen Fang
• Lequan Yu
• Guodong Li

The depth of neural networks is a critical factor for their capability, with deeper models often demonstrating superior performance. Motivated by this, significant efforts have been made to enhance layer aggregation - reusing information from previous layers to better extract features at the current layer, to improve the representational power of deep neural networks. However, previous works have primarily addressed this problem from a discrete-state perspective which is not suitable as the number of network layers grows. This paper novelly treats the outputs from layers as states of a continuous process and considers leveraging the state space model (SSM) to design the aggregation of layers in very deep neural networks. Moreover, inspired by its advancements in modeling long sequences, the Selective State Space Models (S6) is employed to design a new module called Selective State Space Model Layer Aggregation (S6LA). This module aims to combine traditional CNN or transformer architectures within a sequential framework, enhancing the representational capabilities of state-of-the-art vision networks. Extensive experiments show that S6LA delivers substantial improvements in both image classification and detection tasks, highlighting the potential of integrating SSMs with contemporary deep learning techniques.

AAAI Conference 2025 Conference Paper

GapMatch: Bridging Instance and Model Perturbations for Enhanced Semi-Supervised Medical Image Segmentation

• Wei Huang
• Lei Zhang
• Zizhou Wang
• Yan Wang

Medical image segmentation provides detailed understanding and aids in diagnosis, treatment planning, and monitoring of diseases. Due to the high cost of acquiring labeled data in the field of medical image analysis, semi-supervised segmentation methods have garnered increasing attention. Benefiting from their simplicity and effectiveness, consistency regularization-based methods have emerged as a significant research focus by utilizing perturbations. However, existing methods typically consider perturbation strategies from only a single perspective: either instance perturbation or model perturbation, thus ignoring the potential benefit of effectively combining both. In response, we propose a unified perturbation framework named GapMatch, which bridges instance and model perturbations to broaden the perturbation space and employs dual perturbation to impose consistency regularization on the model. Specifically, GapMatch involves using instance perturbation to update the decision boundary and model perturbation to further optimize it. These two steps mutually reinforce each other in an iterative manner, effectively pushing the decision boundary towards low-density regions while maximizing the class margin. Extensive experimental results on two popular medical image benchmarks demonstrate the effectiveness and generality of the proposed method.

NeurIPS Conference 2025 Conference Paper

Generalization Bound of Gradient Flow through Training Trajectory and Data-dependent Kernel

• Yilan Chen
• Zhichao Wang
• Wei Huang
• Andi Han
• Taiji Suzuki
• Arya Mazumdar

Gradient-based optimization methods have shown remarkable empirical success, yet their theoretical generalization properties remain only partially understood. In this paper, we establish a generalization bound for gradient flow that aligns with the classical Rademacher complexity bounds for kernel methods—specifically those based on the RKHS norm and kernel trace—through a data-dependent kernel called the loss path kernel (LPK). Unlike static kernels such as NTK, the LPK captures the entire training trajectory, adapting to both data and optimization dynamics, leading to tighter and more informative generalization guarantees. Moreover, the bound highlights how the norm of the training loss gradients along the optimization trajectory influences the final generalization performance. The key technical ingredients in our proof combine stability analysis of gradient flow with uniform convergence via Rademacher complexity. Our bound recovers existing kernel regression bounds for overparameterized neural networks and shows the feature learning capability of neural networks compared to kernel methods. Numerical experiments on real-world datasets validate that our bounds correlate well with the true generalization gap.

NeurIPS Conference 2025 Conference Paper

How Does Label Noise Gradient Descent Improve Generalization in the Low SNR Regime?

• Wei Huang
• Andi Han
• Yujin Song
• Yilan Chen
• Denny Wu
• Difan Zou
• Taiji Suzuki

The capacity of deep learning models is often large enough to both learn the underlying statistical signal and overfit to noise in the training set. This noise memorization can be harmful especially for data with a low signal-to-noise ratio (SNR), leading to poor generalization. Inspired by prior observations that label noise provides implicit regularization that improves generalization, in this work, we investigate whether introducing label noise to the gradient updates can enhance the test performance of neural network (NN) in the low SNR regime. Specifically, we consider training a two-layer NN with a simple label noise gradient descent (GD) algorithm, in an idealized signal-noise data setting. We prove that adding label noise during training suppresses noise memorization, preventing it from dominating the learning process; consequently, label noise GD enjoys rapid signal growth while the overfitting remains controlled, thereby achieving good generalization despite the low SNR. In contrast, we also show that NN trained with standard GD tends to overfit to noise in the same low SNR setting and establish a non-vanishing lower bound on its test error, thus demonstrating the benefit of introducing label noise in gradient-based training.

JBHI Journal 2025 Journal Article

In Vitro Diagnosis of Parkinson's Disease Based on Facial Expression and Behavioral Gait Data

• Yinxuan Xu
• Yintao Zhou
• Zhengyu Li
• Jing Huang
• Wei Huang

Parkinson's disease (PD) is characterized by incurable, rapid progression, and severe disability, severely impacting the lives of patients and their families. With an aging population, the need for early detection of PD is increasing. In vitro diagnosis has attracted attention because of its non-invasiveness and low cost, but there are some problems with the existing methods: 1) facial expression diagnosis has little training data; 2) gait diagnosis requires specialized equipment and acquisition environment, which is poorly generalizable; 3) a single modality is easy to miss the diagnosis; and 4) multimodal diagnostic methods are not universally applicable. To address the above issues, we propose a novel multimodal in vitro diagnostic method for PD based on facial expression and behavioral gait. The method uses a lightweight deep learning model for feature extraction and feature fusion to improve diagnostic accuracy and ease of use. Meanwhile, we have established the largest multimodal PD data set in collaboration with hospitals and conducted a large number of experiments to verify the effectiveness of the method.

NeurIPS Conference 2025 Conference Paper

Less is More: an Attention-free Sequence Prediction Modeling for Offline Embodied Learning

• Wei Huang
• Jianshu Zhang
• Leiyu Wang
• Heyue Li
• Luoyi Fan
• Yichen Zhu
• Nanyang Ye
• Qinying Gu

Offline reinforcement learning (offline RL) is increasingly approached as a sequence modeling task, with methods leveraging advanced architectures like Transformers to capture trajectory dependencies. Despite significant progress, the mechanisms underlying their effectiveness and limitations remain insufficiently understood. We conduct a thorough analysis on the representative Decision Transformer (DT) model using an entropy analysis and identify the inconsistencies in state-action-reward ($\langle s, a, R \rangle$) distributions causing attention ``dispersal". To address this, we propose a hierarchical framework that decomposes sequence modeling into intra-step relational modeling—handled by a Token Merger that fuses each $\langle s, a, R \rangle$ triplet—and inter-step modeling—handled by a Token Mixer across timesteps. We investigate several Token Merger designs and validate their effectiveness across various offline RL methods. Furthermore, our theoretical analysis and experimental results suggest that while Token Mixers are important, lightweight architecture can also achieve even better performance to more complex ones. We therefore propose a parameter-free Average Pooling Token Mixer, which, combined with a convolutional Token Merger, forms our final model, Decision HiFormer (DHi). DHi achieves a \textbf{73. 6\%} improvement in inference speed and an \textbf{9. 3\%} gain in policy performance on the D4RL benchmark compared to DT. DHi also generalizes well to real-world robotic manipulation tasks, offering both practical benefits and insights into sequence-based policy design for offline RL. Code and models are public at \href{https: //wei-nijuan. github. io/DecisionHiFormer/}{project page}.

NeurIPS Conference 2025 Conference Paper

MindOmni: Unleashing Reasoning Generation in Vision Language Models with RGPO

• Yicheng Xiao
• Lin Song
• Yukang Chen
• Yingmin Luo
• Yuxin Chen
• Yukang Gan
• Wei Huang
• Xiu Li

Recent text-to-image systems face limitations in handling multimodal inputs and complex reasoning tasks. We introduce MindOmni, a unified multimodal large language model that addresses these challenges by incorporating reasoning generation through reinforcement learning. MindOmni leverages a three-phase training strategy: i) design of a unified vision language model with a decoder-only diffusion module, ii) supervised fine-tuning with Chain-of-Thought (CoT) instruction data, and iii) our proposed Reasoning Generation Policy Optimization (RGPO) algorithm, utilizing multimodal feedback to effectively guide policy updates. Experimental results demonstrate that MindOmni outperforms existing models, achieving impressive performance on both understanding and generation benchmarks, meanwhile showcasing advanced fine-grained reasoning generation capabilities, especially with mathematical reasoning instruction. All codes will be made public.

ICML Conference 2025 Conference Paper

Multinoulli Extension: A Lossless Yet Effective Probabilistic Framework for Subset Selection over Partition Constraints

• Qixin Zhang 0001
• Wei Huang
• Can Jin
• Puning Zhao
• Yao Shu
• Li Shen 0008
• Dacheng Tao

Identifying the most representative subset for a close-to-submodular objective while satisfying the predefined partition constraint is a fundamental task with numerous applications in machine learning. However, the existing distorted local-search methods are often hindered by their prohibitive query complexities and the rigid requirement for prior knowledge of difficult-to-obtain structural parameters. To overcome these limitations, we introduce a novel algorithm titled Multinoulli-SCG, which not only is parameter-free, but also can achieve the same approximation guarantees as the distorted local-search methods with significantly fewer function evaluations. The core of our Multinoulli-SCG algorithm is an innovative continuous-relaxation framework named Multinoulli Extension( ME ), which can effectively convert the discrete subset selection problem subject to partition constraints into a solvable continuous maximization focused on learning the optimal multinoulli priors across the considered partition. In sharp contrast with the well-established multi-linear extension for submodular subset selection, a notable advantage of our proposed ME is its intrinsic capacity to provide a lossless rounding scheme for any set function. Finally, we validate the practical efficacy of our proposed algorithms by applying them to video summarization, bayesian A-optimal design and coverage maximization.

ICML Conference 2025 Conference Paper

Projection Pursuit Density Ratio Estimation

• Meilin Wang
• Wei Huang
• Mingming Gong
• Zheng Zhang

Density ratio estimation (DRE) is a paramount task in machine learning, for its broad applications across multiple domains, such as covariate shift adaptation, causal inference, independence tests and beyond. Parametric methods for estimating the density ratio possibly lead to biased results if models are misspecified, while conventional non-parametric methods suffer from the curse of dimensionality when the dimension of data is large. To address these challenges, in this paper, we propose a novel approach for DRE based on the projection pursuit (PP) approximation. The proposed method leverages PP to mitigate the impact of high dimensionality while retaining the model flexibility needed for the accuracy of DRE. We establish the consistency and the convergence rate for the proposed estimator. Experimental results demonstrate that our proposed method outperforms existing alternatives in various applications.

NeurIPS Conference 2025 Conference Paper

Rethinking Out-of-Distribution Detection and Generalization with Collective Behavior Dynamics

• Zhenbin Wang
• Lei Zhang
• Wei Huang
• Zhao Zhang
• Zizhou Wang

Out-of-distribution (OOD) problems commonly occur when models process data with a distribution significantly deviates from the in-distribution (InD) training data. In this paper, we hypothesize that a $\textit{field}$ or $\textit{potential}$ more essential than features exists, and features are not the ultimate essence of the data but rather manifestations of them during training. we investigate OOD problems from the perspective of collective behavior dynamics. With this in mind, we first treat the output of the feature extractor as charged particles and investigate their collective behavior dynamics within a self-consistent electric field. Then, to characterize the relationship between OOD problems and dynamical equations, we introduce the $\textit{basin of attraction}$ and prove that its boundary can be represented as the zero level set of a differentiable function of the potential, $\textit{i. e. }$, the spatial integral of field. We further demonstrate that: $\textit{i)}$ InD and OOD inputs can be effectively separated based on whether they are steady state solutions for specific field conditions, enabling robust OOD detection and outperforming prior methods over three benchmarks. $\textit{ii)}$ the generalization capability correlates positively with the basin of attraction. By analyzing the dynamics of perturbations, we propose that the potential is well-characterized by a Fourier-domain form of the Poisson equation. Evaluated on six benchmark datasets, our method rivals the SoTA approaches for OOD generalization and can be seamlessly integrated with them to deliver additional gains.

AAAI Conference 2025 Conference Paper

Risk Controlled Image Retrieval

• Kaiwen Cai
• Chris Xiaoxuan Lu
• Xingyu Zhao
• Wei Huang
• Xiaowei Huang

Most image retrieval research prioritizes improving predictive performance, often overlooking situations where the reliability of predictions is equally important. The gap between model performance and reliability requirements highlights the need for a systematic approach to analyze and address the risks associated with image retrieval. Uncertainty quantification technique can be applied to mitigate this issue by assessing uncertainty for retrieval sets, but it provides only a heuristic estimate of uncertainty rather than a guarantee. To address these limitations, we present Risk Controlled Image Retrieval (RCIR), which generates retrieval sets with coverage guarantee, i.e., retrieval sets that are guaranteed to contain the true nearest neighbors with a predefined probability. RCIR can be easily integrated with existing uncertainty-aware image retrieval systems, agnostic to data distribution and model selection. To the best of our knowledge, this is the first work that provides coverage guarantees to image retrieval. The validity and efficiency of RCIR are demonstrated on four real-world datasets: CAR-196, CUB-200, Pittsburgh, and ChestX-Det.

NeurIPS Conference 2025 Conference Paper

Scaling Diffusion Transformers Efficiently via $\mu$P

• Chenyu Zheng
• Xinyu Zhang
• Rongzhen Wang
• Wei Huang
• Zhi Tian
• Weilin Huang
• Jun Zhu
• Chongxuan Li

Diffusion Transformers have emerged as the foundation for vision generative models, but their scalability is limited by the high cost of hyperparameter (HP) tuning at large scales. Recently, Maximal Update Parametrization ($\mu$P) was proposed for vanilla Transformers, which enables stable HP transfer from small to large language models, and dramatically reduces tuning costs. However, it remains unclear whether $\mu$P of vanilla Transformers extends to diffusion Transformers, which differ architecturally and objectively. In this work, we generalize $\mu$P to diffusion Transformers and validate its effectiveness through large-scale experiments. First, we rigorously prove that $\mu$P of mainstream diffusion Transformers, including DiT, U-ViT, PixArt-$\alpha$, and MMDiT, aligns with that of the vanilla Transformer, enabling the direct application of existing $\mu$P methodologies. Leveraging this result, we systematically demonstrate that DiT-$\mu$P enjoys robust HP transferability. Notably, DiT-XL-2-$\mu$P with transferred learning rate achieves 2. 9$\times$ faster convergence than the original DiT-XL-2. Finally, we validate the effectiveness of $\mu$P on text-to-image generation by scaling PixArt-$\alpha$ from 0. 04B to 0. 61B and MMDiT from 0. 18B to 18B. In both cases, models under $\mu$P outperform their respective baselines while requiring small tuning cost—only 5. 5% of one training run for PixArt-$\alpha$ and 3% of consumption by human experts for MMDiT-18B. \textit{These results establish $\mu$P as a principled and efficient framework for scaling diffusion Transformers}.

NeurIPS Conference 2025 Conference Paper

Scaling RL to Long Videos

• Yukang Chen
• Wei Huang
• Baifeng Shi
• Qinghao Hu
• Hanrong Ye
• Ligeng Zhu
• Zhijian Liu
• Pavlo Molchanov

We introduce a full-stack framework that scales up reasoning in vision-language models (VLMs) to long videos, leveraging reinforcement learning. We address the unique challenges of long video reasoning by integrating three critical components: (1) a large-scale dataset, LongVideo-Reason, comprising 104K long video QA pairs with high-quality reasoning annotations across diverse domains such as sports, games, and vlogs; (2) a two-stage training pipeline that extends VLMs with chain-of-thought supervised fine-tuning (CoT-SFT) and reinforcement learning (RL); and (3) a training infrastructure for long video RL, named Multi-modal Reinforcement Sequence Parallelism (MR-SP), which incorporates sequence parallelism and a vLLM-based engine tailored for long video, using cached video embeddings for efficient rollout and prefilling. In our experiments, LongVILA-R1-7B achieves strong performance on video benchmarks, reaching 65. 1% and 71. 1% accuracy on VideoMME without and with subtitles, respectively, and consistently outperforming LongVILA-7B across multiple benchmarks. Moreover, LongVILA-R1-7B supports processing up to 8, 192 video frames per video, and configurable FPS settings. Notably, our MR-SP system achieves up to 2. 1x speedup on long video RL training. In addition, we release our training system for public availability that supports RL training on various modalities (video, text, and audio), various models (VILA and Qwen series), and even image and video generation models. On a single A100 node (8 GPUs), it supports RL training on hour-long videos (e. g. , 3, 600 frames). Code and models are available at https: //github. com/NVlabs/Long-RL

EAAI Journal 2025 Journal Article

Symmetric non-negative matrix factorization-based deep representation algorithm for multi-view clustering

• Ping Deng
• Xinying Zhou
• Ji Xu
• Wei Huang
• Jie Wang
• Dexian Wang
• Tianrui Li

Symmetric Non-negative Matrix Factorization (SNMF) shows significant advantages in clustering task due to its unique mathematical properties. However, it still has several key limitations: (1) the single optimization scheme of traditional multiplicative update rule limits the flexibility of the algorithm; (2) linear factorization leads to insufficient representation ability for complex nonlinear features; (3) lack of learning rate guidance mechanism. These factors together constrain the algorithm representation learning ability in complex data. To address these issues, this paper proposes a SNMF-based Deep Representation algorithm for Multi-view Clustering (SNDRMvC). First, the matrix elements are decoupled, and the stochastic gradient descent as well as nonlinear activation function are used to implement non-negative matrix update. Then, based on the corresponding gradients of the elements and nonlinear function, the neural network learning mechanism is introduced into the SNMF update rule to construct a novel framework SNMF-based deep representation network for optimizing SNMF. This network aims to update the elements in the low-dimensional matrix of each view and fuse the low-dimensional matrices of multiple views to derive a consensus matrix. Finally, extensive experiments conducted on several public datasets demonstrate that the proposed algorithm exhibits notable advantages in clustering performance. We provide the code at: https: //github. com/Code706/SNDRMvC.

AAAI Conference 2025 Conference Paper

TIME-FS: Joint Learning of Tensorial Incomplete Multi-View Unsupervised Feature Selection and Missing-View Imputation

• Yanyong Huang
• Minghui Lu
• Wei Huang
• Xiuwen Yi
• Tianrui Li

Multi-view unsupervised feature selection (MUFS) has received considerable attention in recent years. Existing MUFS methods for processing unlabeled incomplete multi-view data, where some samples are missing in certain views, first impute the missing values and then perform feature selection on the completed dataset. However, treating imputation and feature selection as two separate processes overlooks their potential interactions. The graph-guided local structure gleaned from feature selection can aid in imputation, which in turn can enhance the feature selection performance. Additionally, most similarity graph-based MUFS methods suffer from high computational costs. To address these problems, we propose a novel MUFS method, termed Tensorial Incomplete Multi-view unsupErvised Feature Selection (TIME-FS). TIME-FS unifies missing value recovery, discriminative feature selection, and low-dimensional representation learning within a joint framework through matrix decomposition. Then, TIME-FS conducts CP decomposition on tensor data formed by the low-dimensional representations of different views to learn a consistent anchor graph across views and a view-preference weight matrix, both of which simultaneously guide missing view imputation and feature selection. Furthermore, an efficient algorithm with low time complexity and rapid convergence is proposed to solve TIME-FS. Extensive experimental results demonstrate the effectiveness and efficiency of TIME-FS over state-of-the-art methods.

NeurIPS Conference 2025 Conference Paper

Towards Unsupervised Training of Matching-based Graph Edit Distance Solver via Preference-aware GAN

• Wei Huang
• Hanchen Wang
• Dong Wen
• SHAOZHEN MA
• Wenjie Zhang
• Xuemin Lin

Graph Edit Distance (GED) is a fundamental graph similarity metric widely used in various applications. However, computing GED is an NP-hard problem. Recent state-of-the-art hybrid GED solver has shown promising performance by formulating GED as a bipartite graph matching problem, then leveraging a generative diffusion model to predict node matching between two graphs, from which both the GED and its corresponding edit path can be extracted using a traditional algorithm. However, such methods typically rely heavily on ground-truth supervision, where the ground-truth node matchings are often costly to obtain in real-world scenarios. In this paper, we propose GEDRanker, a novel unsupervised GAN-based framework for GED computation. Specifically, GEDRanker consists of a matching-based GED solver and introduces an interpretable preference-aware discriminator. By leveraging preference signals over different node matchings derived from edit path lengths, the discriminator can guide the matching-based solver toward generating high-quality node matching without the need for ground-truth supervision. Extensive experiments on benchmark datasets demonstrate that our GEDRanker enables the matching-based GED solver to achieve near-optimal solution quality without any ground-truth supervision.

NeurIPS Conference 2025 Conference Paper

Trained Mamba Emulates Online Gradient Descent in In-Context Linear Regression

• Jiarui Jiang
• Wei Huang
• Miao Zhang
• Taiji Suzuki
• Liqiang Nie

State-space models (SSMs), particularly Mamba, emerge as an efficient Transformer alternative with linear complexity for long-sequence modeling. Recent empirical works demonstrate Mamba's in-context learning (ICL) capabilities competitive with Transformers, a critical capacity for large foundation models. However, theoretical understanding of Mamba’s ICL remains limited, restricting deeper insights into its underlying mechanisms. Even fundamental tasks such as linear regression ICL, widely studied as a standard theoretical benchmark for Transformers, have not been thoroughly analyzed in the context of Mamba. To address this gap, we study the training dynamics of Mamba on the linear regression ICL task. By developing novel techniques tackling non-convex optimization with gradient descent related to Mamba's structure, we establish an exponential convergence rate to ICL solution, and derive a loss bound that is comparable to Transformer's. Importantly, our results reveal that Mamba can perform a variant of \textit{online gradient descent} to learn the latent function in context. This mechanism is different from that of Transformer, which is typically understood to achieve ICL through gradient descent emulation. The theoretical results are verified by experimental simulation.

ICLR Conference 2024 Conference Paper

A Variational Framework for Estimating Continuous Treatment Effects with Measurement Error

• Erdun Gao
• Howard D. Bondell
• Wei Huang
• Mingming Gong

Estimating treatment effects has numerous real-world applications in various fields, such as epidemiology and political science. While much attention has been devoted to addressing the challenge using fully observational data, there has been comparatively limited exploration of this issue in cases when the treatment is not directly observed. In this paper, we tackle this problem by developing a general variational framework, which is flexible to integrate with advanced neural network-based approaches, to identify the average dose-response function (ADRF) with the continuously valued error-contaminated treatment. Our approach begins with the formulation of a probabilistic data generation model, treating the unobserved treatment as a latent variable. In this model, we leverage a learnable density estimation neural network to derive its prior distribution conditioned on covariates. This module also doubles as a generalized propensity score estimator, effectively mitigating selection bias arising from observed confounding variables. Subsequently, we calculate the posterior distribution of the treatment, taking into account the observed measurement and outcome. To mitigate the impact of treatment error, we introduce a re-parametrized treatment value, replacing the error-affected one, to make more accurate predictions regarding the outcome. To demonstrate the adaptability of our framework, we incorporate two state-of-the-art ADRF estimation methods and rigorously assess its efficacy through extensive simulations and experiments using semi-synthetic data.

JBHI Journal 2024 Journal Article

Adaptive Annotation Correlation Based Multi-Annotation Learning for Calibrated Medical Image Segmentation

• Wei Huang
• Lei Zhang
• Xin Shu
• Zizhou Wang
• Zhang Yi

Medical image segmentation is a fundamental task in many clinical applications, yet current automated segmentation methods rely heavily on manual annotations, which are inherently subjective and prone to annotation bias. Recently, modeling annotator preference has garnered great interest, and several methods have been proposed in the past two years. However, the existing methods completely ignore the potential correlation between annotations, such as complementary and discriminative information. In this work, the A daptive annotation C orrela T ion based mult I -ann O tation Lear N ing ( ACTION ) method is proposed for calibrated medical image segmentation. ACTION employs consensus feature learning and dynamic adaptive weighting to leverage complementary information across annotations and emphasize discriminative information within each annotation based on their correlations, respectively. Meanwhile, memory accumulation-replay is proposed to accumulate the prior knowledge and integrate it into the model to enable the model to accommodate the multi-annotation setting. Two medical image benchmarks with different modalities are utilized to evaluate the performance of ACTION, and extensive experimental results demonstrate that it achieves superior performance compared to several state-of-the-art methods.

JBHI Journal 2024 Journal Article

Auto Diagnosis of Parkinson's Disease Via a Deep Learning Model Based on Mixed Emotional Facial Expressions

• Wei Huang
• Wenqiang Xu
• Renjie Wan
• Peng Zhang
• Yufei Zha
• Meng Pang

Parkinson's disease (PD) is a common degenerative disease of the nervous system in the elderly. The early diagnosis of PD is very important for potential patients to receive prompt treatment and avoid the aggravation of the disease. Recent studies have found that PD patients always suffer from emotional expression disorder, thus forming the characteristics of “masked faces”. Based on this, we thus propose an auto PD diagnosis method based on mixed emotional facial expressions in the paper. Specifically, the proposed method is cast into four steps: Firstly, we synthesize virtual face images containing six basic expressions (i. e. , anger, disgust, fear, happiness, sadness, and surprise) via generative adversarial learning, in order to approximate the premorbid expressions of PD patients; Secondly, we design an effective screening scheme to assess the quality of the above synthesized facial expression images and then shortlist the high-quality ones; Thirdly, we train a deep feature extractor accompanied with a facial expression classifier based on the mixture of the original facial expression images of the PD patients, the high-quality synthesized facial expression images of PD patients, and the normal facial expression images from other public face datasets; Finally, with the well-trained deep feature extractor, we thus adopt it to extract the latent expression features for six facial expression images of a potential PD patient to conduct PD/non-PD prediction. To show real-world impacts, we also collected a new facial expression dataset of PD patients in collaboration with a hospital. Extensive experiments are conducted to validate the effectiveness of the proposed method for PD diagnosis and facial expression recognition.

ICRA Conference 2024 Conference Paper

Design and Implementation of A Robotized Hand-held Dissector for Endoscopic Pulmonary Endarterectomy

• Runfeng Zhu
• Xilong Hou 0001
• Wei Huang
• Lei Du
• Zhong Wu
• Hongbin Liu 0001
• Henry Kar Hang Chu
• Qingxiang Zhao

Severe chronic pulmonary endarterectomy needs a dissector to delicately remove proliferative intima located in the depth of the pulmonary artery. This work proposed a novel endoscopic robotized steerable dissector for this surgery, enabling easier access to curved deep artery branches. The handheld surgical dissector also provides suction and visualization for surgeons to enhance effectiveness. The steerable section is a cable-driven hinged structure, and through an antagonistic mechanism regulating the cable tension, the overall stiffness is adjusted to adapt various surroundings. The mapping between actuation space and shape configuration and tip force estimation model are respectively established for further closed-loop control scheme, achieving adaptive positioning and safe surgery. Experiments first demonstrate the feasibility of the proposed models and ex vitro trials validated the usage and effectiveness of the robotized dissector.

ICML Conference 2024 Conference Paper

Diffusion Models Demand Contrastive Guidance for Adversarial Purification to Advance

• Mingyuan Bai
• Wei Huang
• Tenghui Li
• Andong Wang
• Junbin Gao
• Cesar F. Caiafa
• Qibin Zhao

In adversarial defense, adversarial purification can be viewed as a special generation task with the purpose to remove adversarial attacks and diffusion models excel in adversarial purification for their strong generative power. With different predetermined generation requirements, various types of guidance have been proposed, but few of them focuses on adversarial purification. In this work, we propose to guide diffusion models for adversarial purification using contrastive guidance. We theoretically derive the proper noise level added in the forward process diffusion models for adversarial purification from a feature learning perspective. For the reverse process, it is implied that the role of contrastive loss guidance is to facilitate the evolution towards the signal direction. From the theoretical findings and implications, we design the forward process with the proper amount of Gaussian noise added and the reverse process with the gradient of contrastive loss as the guidance of diffusion models for adversarial purification. Empirically, extensive experiments on CIFAR-10, CIFAR-100, the German Traffic Sign Recognition Benchmark and ImageNet datasets with ResNet and WideResNet classifiers show that our method outperforms most of current adversarial training and adversarial purification methods by a large improvement.

EAAI Journal 2024 Journal Article

Dilated-Windows-based Vision Transformer with Efficient-Suppressive-self-attention for insect pests classification

• Zhenzhe Hechen
• Wei Huang
• Le Yin
• Wenjing Xie
• Yixin Zhao

Transformers with long-range dependency and data specificity act as an effective means of classifying insect pests in agricultural engineering. Although many methods have been proposed to confine the range of self-attention within a local region to reduce the computation complexity, none of them can reduce the number of model parameters. Moreover, the self-attention mechanism usually causes query tokens to focus excessively on image patches, which limits the effective receptive field and the long-range dependence. To address these issues, this paper establishes a novel Dilated-Windows-based Vision Transformer with Efficient-Suppressive-self-attention (DWViT-ES) architecture, which includes efficient-self-attention (ESA), dilated window (DW), and suppressive-self-attention (SSA) as its core components. The ESA simplifies the successive linear Transformations to reduce the number of model parameters and computational costs. Meanwhile, the DW and SSA expand the effective receptive field of self-attention mechanism to prevent query tokens from focusing on similar and close regions, thereby preventing the loss of useful information. Finally, experiments show that the DWViT-ES only has 19. 6 M parameters and 3. 5G FLOPs (over 20% reductions vs. 19. 6 M and 4. 5G of Swin-T). Meanwhile, the DWViT-ES training from scratch has 71. 6% top-1 accuracy on the IP102 dataset (2. 4% absolute improvement of Swin-T); after fine-tuning on Imagenet-1K, the DWViT-ES achieves 76. 0% and 78. 7% top-1 accuracy on IP102 and CPB (0. 1% and 0. 9% absolute improvement of Swin-T), respectively. Meanwhile, practical deployment on a mobile-embedded device is presented, which validates the feasibility of the DWViT-ES.

TIST Journal 2024 Journal Article

DNSRF: Deep Network-based Semi-NMF Representation Framework

• Dexian Wang
• Tianrui Li
• Ping Deng
• Zhipeng Luo
• Pengfei Zhang
• Keyu Liu
• Wei Huang

Representation learning is an important topic in machine learning, pattern recognition, and data mining research. Among many representation learning approaches, semi-nonnegative matrix factorization (SNMF) is a frequently-used one. However, a typical problem of SNMF is that usually there is no learning rate guidance during the optimization process, which often leads to a poor representation ability. To overcome this limitation, we propose a very general representation learning framework (DNSRF) that is based on a deep neural net. Essentially, the parameters of the deep net used to construct the DNSRF algorithms are obtained by matrix element update. In combination with different activation functions, DNSRF can be implemented in various ways. In our experiments, we tested nine instances of our DNSRF framework on six benchmark datasets. In comparison with other state-of-the-art methods, the results demonstrate the superior performance of our framework, which is thus shown to have a great representation ability.

EAAI Journal 2024 Journal Article

DRNAS: Differentiable RBF neural architecture search method considering computation load in adaptive control

• Ruichen Ming
• XiaoXiong Liu
• Yu Li
• Wei Huang
• Weiguo Zhang

In this work, we investigated an online differential neural network search control algorithm using a backstepping method with a radial basis function (RBF) neural network (NN) framework. In this approach, we mainly focused on searching a neural network architecture with optimal control performance and optimal computation load by learning NN parameters among a finite number of RBF NNs with different architectures. The previous works on RBFNN and backstepping methods mainly considered the control performance of systems, and the computation load limitations of control computers were rarely considered. In this paper, we herein propose a differentiable RBF neural architecture search (DRNAS) method. First, we built a hypernetwork and constructed an appropriate optimization objective function with information of a tracking error and a computation load. This hypernetwork consists of different networks with weight parameters. Then, through backpropagation and based on the gradient descent method, we updated the parameters of the hypernetwork and determined the optimal RBF NN architecture in the search space. Finally, we performed simulations to verify the effectiveness of the proposed method, where we designed an RBF NN adaptive backstepping controller for aircraft pitch rate dynamics and used the DRNAS method to train the hypernetwork based on different mission scenarios. The simulation results verified that the proposed method can effectively balance the controller’s tracking capability with its computation load.

AAAI Conference 2024 Conference Paper

Earthfarsser: Versatile Spatio-Temporal Dynamical Systems Modeling in One Model

• Hao Wu
• Yuxuan Liang
• Wei Xiong
• Zhengyang Zhou
• Wei Huang
• Shilong Wang
• Kun Wang

Efficiently modeling spatio-temporal (ST) physical processes and observations presents a challenging problem for the deep learning community. Many recent studies have concentrated on meticulously reconciling various advantages, leading to designed models that are neither simple nor practical. To address this issue, this paper presents a systematic study on existing shortcomings faced by off-the-shelf models, including lack of local fidelity, poor prediction performance over long time-steps, low scalability, and inefficiency. To systematically address the aforementioned problems, we propose an EarthFarseer, a concise framework that combines parallel local convolutions and global Fourier-based transformer architectures, enabling dynamically capture the local-global spatial interactions and dependencies. EarthFarseer also incorporates a multi-scale fully convolutional and Fourier architectures to efficiently and effectively capture the temporal evolution. Our proposal demonstrates strong adaptability across various tasks and datasets, with fast convergence and better local fidelity in long time-steps predictions. Extensive experiments and visualizations over eight human society physical and natural physical datasets demonstrates the state-of-the-art performance of EarthFarseer. We release our code at https://github.com/easylearningscores/EarthFarseer.

IJCAI Conference 2024 Conference Paper

Enhancing Dual-Target Cross-Domain Recommendation with Federated Privacy-Preserving Learning

• Zhenghong Lin
• Wei Huang
• Hengyu Zhang
• Jiayu Xu
• Weiming Liu
• Xinting Liao
• Fan Wang
• Shiping Wang

Recently, dual-target cross-domain recommendation (DTCDR) has been proposed to alleviate the data sparsity problem by sharing the common knowledge across domains simultaneously. However, existing methods often assume that personal data containing abundant identifiable information can be directly accessed, which results in a controversial privacy leakage problem of DTCDR. To this end, we introduce the P2DTR framework, a novel approach in DTCDR while protecting private user information. Specifically, we first design a novel inter-client knowledge extraction mechanism, which exploits the private set intersection algorithm and prototype-based federated learning to enable collaboratively modeling among multiple users and a server. Furthermore, to improve the recommendation performance based on the extracted common knowledge across domains, we proposed an intra-client enhanced recommendation, consisting of a constrained dominant set (CDS) propagation mechanism and dual-recommendation module. Extensive experiments on real-world datasets validate that our proposed P2DTR framework achieves superior utility under a privacy-preserving guarantee on both domains.

NeurIPS Conference 2024 Conference Paper

Federated Learning from Vision-Language Foundation Models: Theoretical Analysis and Method

• Bikang Pan
• Wei Huang
• Ye Shi

Integrating pretrained vision-language foundation models like CLIP into federated learning has attracted significant attention for enhancing generalization across diverse tasks. Typically, federated learning of vision-language models employs prompt learning to reduce communication and computational costs, i. e. , prompt-based federated learning. However, there is limited theoretical analysis to understand the performance of prompt-based federated learning. In this work, we construct a theoretical analysis framework for prompt-based federated learning via feature learning theory. Specifically, we monitor the evolution of signal learning and noise memorization in prompt-based federated learning, demonstrating that performance can be assessed by the ratio of task-relevant to task-irrelevant coefficients. Furthermore, we draw an analogy between income and risk in portfolio optimization and the task-relevant and task-irrelevant terms in feature learning. Leveraging inspiration from portfolio optimization that combining two independent assets will maintain the income while reducing the risk, we introduce two prompts: global prompt and local prompt to construct a prompt portfolio to balance the generalization and personalization. Consequently, we showed the performance advantage of the prompt portfolio and derived the optimal mixing coefficient. These theoretical claims have been further supported by empirical experiments.

NeurIPS Conference 2024 Conference Paper

Identifiability Analysis of Linear ODE Systems with Hidden Confounders

• Yuanyuan Wang
• Biwei Huang
• Wei Huang
• Xi Geng
• Mingming Gong

The identifiability analysis of linear Ordinary Differential Equation (ODE) systems is a necessary prerequisite for making reliable causal inferences about these systems. While identifiability has been well studied in scenarios where the system is fully observable, the conditions for identifiability remain unexplored when latent variables interact with the system. This paper aims to address this gap by presenting a systematic analysis of identifiability in linear ODE systems incorporating hidden confounders. Specifically, we investigate two cases of such systems. In the first case, latent confounders exhibit no causal relationships, yet their evolution adheres to specific functional forms, such as polynomial functions of time $t$. Subsequently, we extend this analysis to encompass scenarios where hidden confounders exhibit causal dependencies, with the causal structure of latent variables described by a Directed Acyclic Graph (DAG). The second case represents a more intricate variation of the first case, prompting a more comprehensive identifiability analysis. Accordingly, we conduct detailed identifiability analyses of the second system under various observation conditions, including both continuous and discrete observations from single or multiple trajectories. To validate our theoretical results, we perform a series of simulations, which support and substantiate our findings.

JMLR Journal 2024 Journal Article

Identifiability and Asymptotics in Learning Homogeneous Linear ODE Systems from Discrete Observations

• Yuanyuan Wang
• Wei Huang
• Mingming Gong
• Xi Geng
• Tongliang Liu
• Kun Zhang
• Dacheng Tao

Ordinary Differential Equations (ODEs) have recently gained a lot of attention in machine learning. However, the theoretical aspects, for example, identifiability and asymptotic properties of statistical estimation are still obscure. This paper derives a sufficient condition for the identifiability of homogeneous linear ODE systems from a sequence of equally-spaced error-free observations sampled from a single trajectory. When observations are disturbed by measurement noise, we prove that under mild conditions, the parameter estimator based on the Nonlinear Least Squares (NLS) method is consistent and asymptotic normal with $n^{-1/2}$ convergence rate. Based on the asymptotic normality property, we construct confidence sets for the unknown system parameters and propose a new method to infer the causal structure of the ODE system, that is, inferring whether there is a causal link between system variables. Furthermore, we extend the results to degraded observations, including aggregated and time-scaled ones. To the best of our knowledge, our work is the first systematic study of the identifiability and asymptotic properties in learning linear ODE systems. We also construct simulations with various system dimensions to illustrate the established theoretical results. [abs] [ pdf ][ bib ] &copy JMLR 2024. ( edit, beta )

EAAI Journal 2024 Journal Article

Joint Cauchy dictionary learning and graph learning for unsupervised feature selection

• Jing-Xin Liu
• Qing-Peng Zeng
• Jian-Sheng Wu
• Wei Huang

Due to its efficiency and flexibility, many unsupervised feature selection models based on dictionary learning have been proposed to select prominent features for improving the accuracy of data clustering and classification in pattern recognition, data mining, and machine learning. However, these models generally focus on how to investigate data distribution but ignore the issue that data may be contaminated by outliers, making the existing models ineffective. Motivated by this, we propose Joint Cauchy Dictionary Learning and Graph Learning for Unsupervised Feature Selection in this paper. This method makes an effort to incorporate dictionary learning, dual representation learning, and adaptive graph learning into a unified framework, so that both the global and local correlation information can be utilized to facilitate feature selection. Specifically, we first propose Cauchy dictionary learning to get high-quality latent representations that are resistant to data outliers. Following the high-quality representations, we re-encode latent features (features of latent representations) into the feature space and get dual representations to quantify the significance of data features in retaining the global correlation information of data. Then, the local similarity structure is adaptively learned in the projection space to quantify the significance of data features in retaining the local correlation information of data. Finally, we design an efficient optimization algorithm to optimize this model and theoretically analyze its convergence and computational complexity. The evaluation results on eight real-world datasets show that our method outperforms several state-of-the-art feature selection methods.

ECAI Conference 2024 Conference Paper

MultiCounter: Multiple Action Agnostic Repetition Counting in Untrimmed Videos

• Yin Tang
• Wei Luo
• Jinrui Zhang
• Wei Huang
• Ruihai Jing
• Deyu Zhang

Multi-instance Repetitive Action Counting (MRAC) aims to estimate the number of repetitive actions performed by multiple instances in untrimmed videos, commonly found in human-centric domains like sports and exercise. In this paper, we propose MultiCounter, a fully end-to-end deep learning framework that enables simultaneous detection, tracking, and counting of repetitive actions of multiple human instances. Specifically, MultiCounter incorporates two novel modules: 1) mixed spatiotemporal interaction for efficient context correlation across consecutive frames, and 2) task-specific heads for accurate perception of periodic boundaries and generalization for action-agnostic human instances. We train MultiCounter on a synthetic dataset called MultiRep that is generated from annotated real-world videos. Experiments on the MultiRep dataset validate the fundamental challenge of MRAC tasks and showcase the superiority of our proposed model. Compared to ByteTrack+RepNet, a solution that combines an advanced tracker with a single repetition counter, MultiCounter substantially improves Period-mAP by 41. 0%, reduces AvgMAE by 58. 6%, and increases AvgOBO 1. 48 times. This sets a new benchmark in the field of MRAC. Moreover, MultiCounter runs in real-time on a commodity GPU server and is insensitive to the number of human instances in a video.

NeurIPS Conference 2024 Conference Paper

On Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capability

• Chenyu Zheng
• Wei Huang
• Rongzhen Wang
• Guoqiang Wu
• Jun Zhu
• Chongxuan Li

Autoregressively trained transformers have brought a profound revolution to the world, especially with their in-context learning (ICL) ability to address downstream tasks. Recently, several studies suggest that transformers learn a mesa-optimizer during autoregressive (AR) pretraining to implement ICL. Namely, the forward pass of the trained transformer is equivalent to optimizing an inner objective function in-context. However, whether the practical non-convex training dynamics will converge to the ideal mesa-optimizer is still unclear. Towards filling this gap, we investigate the non-convex dynamics of a one-layer linear causal self-attention model autoregressively trained by gradient flow, where the sequences are generated by an AR process $x_{t+1} = W x_t$. First, under a certain condition of data distribution, we prove that an autoregressively trained transformer learns $W$ by implementing one step of gradient descent to minimize an ordinary least squares (OLS) problem in-context. It then applies the learned $\widehat{W}$ for next-token prediction, thereby verifying the mesa-optimization hypothesis. Next, under the same data conditions, we explore the capability limitations of the obtained mesa-optimizer. We show that a stronger assumption related to the moments of data is the sufficient and necessary condition that the learned mesa-optimizer recovers the distribution. Besides, we conduct exploratory analyses beyond the first data condition and prove that generally, the trained transformer will not perform vanilla gradient descent for the OLS problem. Finally, our simulation results verify the theoretical results.

NeurIPS Conference 2024 Conference Paper

On the Comparison between Multi-modal and Single-modal Contrastive Learning

• Wei Huang
• Andi Han
• Yongqiang Chen
• Yuan Cao
• Zhiqiang Xu
• Taiji Suzuki

Multi-modal contrastive learning with language supervision has presented a paradigm shift in modern machine learning. By pre-training on a web-scale dataset, multi-modal contrastive learning can learn high-quality representations that exhibit impressive robustness and transferability. Despite its empirical success, the theoretical understanding is still in its infancy, especially regarding its comparison with single-modal contrastive learning. In this work, we introduce a feature learning theory framework that provides a theoretical foundation for understanding the differences between multi-modal and single-modal contrastive learning. Based on a data generation model consisting of signal and noise, our analysis is performed on a ReLU network trained with the InfoMax objective function. Through a trajectory-based optimization analysis and generalization characterization on downstream tasks, we identify the critical factor, which is the signal-to-noise ratio (SNR), that impacts the generalizability in downstream tasks of both multi-modal and single-modal contrastive learning. Through the cooperation between the two modalities, multi-modal learning can achieve better feature learning, leading to improvements in performance in downstream tasks compared to single-modal learning. Our analysis provides a unified framework that can characterize the optimization and generalization of both single-modal and multi-modal contrastive learning. Empirical experiments on both synthetic and real-world datasets further consolidate our theoretical findings.

NeurIPS Conference 2024 Conference Paper

Provable and Efficient Dataset Distillation for Kernel Ridge Regression

• Yilan Chen
• Wei Huang
• Tsui-Wei Weng

Deep learning models are now trained on increasingly larger datasets, making it crucial to reduce computational costs and improve data quality. Dataset distillation aims to distill a large dataset into a small synthesized dataset such that models trained on it can achieve similar performance to those trained on the original dataset. While there have been many empirical efforts to improve dataset distillation algorithms, a thorough theoretical analysis and provable, efficient algorithms are still lacking. In this paper, by focusing on dataset distillation for kernel ridge regression (KRR), we show that one data point per class is already necessary and sufficient to recover the original model's performance in many settings. For linear ridge regression and KRR with surjective feature mappings, we provide necessary and sufficient conditions for the distilled dataset to recover the original model's parameters. For KRR with injective feature mappings of deep neural networks, we show that while one data point per class is not sufficient in general, $k+1$ data points can be sufficient for deep linear neural networks, where $k$ is the number of classes. Our theoretical results enable directly constructing analytical solutions for distilled datasets, resulting in a provable and efficient dataset distillation algorithm for KRR. We verify our theory experimentally and show that our algorithm outperforms previous work such as KIP while being significantly more efficient, e. g. 15840$\times$ faster on CIFAR-100. Our code is available at \href{https: //github. com/Trustworthy-ML-Lab/provable-efficient-dataset-distill-KRR}{GitHub}.

NeurIPS Conference 2024 Conference Paper

Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning

• Dake Bu
• Wei Huang
• Andi Han
• Atsushi Nitanda
• Taiji Suzuki
• Qingfu Zhang
• Hau-San Wong

Transformer-based large language models (LLMs) have displayed remarkable creative prowess and emergence capabilities. Existing empirical studies have revealed a strong connection between these LLMs' impressive emergence abilities and their in-context learning (ICL) capacity, allowing them to solve new tasks using only task-specific prompts without further fine-tuning. On the other hand, existing empirical and theoretical studies also show that there is a linear regularity of the multi-concept encoded semantic representation behind transformer-based LLMs. However, existing theoretical work fail to build up an understanding of the connection between this regularity and the innovative power of ICL. Additionally, prior work often focuses on simplified, unrealistic scenarios involving linear transformers or unrealistic loss functions, and they achieve only linear or sub-linear convergence rates. In contrast, this work provides a fine-grained mathematical analysis to show how transformers leverage the multi-concept semantics of words to enable powerful ICL and excellent out-of-distribution ICL abilities, offering insights into how transformers innovate solutions for certain unseen tasks encoded with multiple cross-concept semantics. Inspired by empirical studies on the linear latent geometry of LLMs, the analysis is based on a concept-based low-noise sparse coding prompt model. Leveraging advanced techniques, this work showcases the exponential 0-1 loss convergence over the highly non-convex training dynamics, which pioneeringly incorporates the challenges of softmax self-attention, ReLU-activated MLPs, and cross-entropy loss. Empirical simulations corroborate the theoretical findings.

EAAI Journal 2024 Journal Article

Resilient automated intersection control of connected vehicles under denial of service attacks

• Jian Gong
• Yuan Zhao
• Jinde Cao
• Wei Huang
• Weijie Chen
• Mahmoud Abdel-Aty

This paper addresses the issue of automated intersection control for connected vehicles in the presence of Denial-of-Service (DoS) attacks. DoS attacks can disrupt the vehicular communication network, leading to increased service time and additional transmission delays, thereby raising a risk of collisions at intersections. To mitigate this problem, we propose a resilient automated intersection control system that ensures safe passage of connected vehicles through non-signalized intersections. First, we establish a communication topology that captures the information transmission between vehicles using a vehicle conflict graph. This topology serves as the basis for designing a distributed control protocol that enables conflict-free cooperation at intersections, taking into account the stochastic delays caused by DoS attacks. To synthesize a resilient controller for the automated intersection control system, we employ the polytopic overapproximation technique based on the real Jordan form. This technique allows us to model the system dynamics and uncertainties, enabling the derivation of a set of sufficient conditions for system stability. By solving a series of linear matrix inequalities, we obtain the controller gains that satisfy these stability conditions. Numerical examples are presented to demonstrate the effectiveness of the proposed control method.

AAAI Conference 2024 Conference Paper

Self-Supervised Multi-Modal Knowledge Graph Contrastive Hashing for Cross-Modal Search

• Meiyu Liang
• Junping Du
• Zhengyang Liang
• Yongwang Xing
• Wei Huang
• Zhe Xue

Deep cross-modal hashing technology provides an effective and efficient cross-modal unified representation learning solution for cross-modal search. However, the existing methods neglect the implicit fine-grained multimodal knowledge relations between these modalities such as when the image contains information that is not directly described in the text. To tackle this problem, we propose a novel self-supervised multi-grained multi-modal knowledge graph contrastive hashing method for cross-modal search (CMGCH). Firstly, in order to capture implicit fine-grained cross-modal semantic associations, a multi-modal knowledge graph is constructed, which represents the implicit multimodal knowledge relations between the image and text as inter-modal and intra-modal semantic associations. Secondly, a cross-modal graph contrastive attention network is proposed to reason on the multi-modal knowledge graph to sufficiently learn the implicit fine-grained inter-modal and intra-modal knowledge relations. Thirdly, a cross-modal multi-granularity contrastive embedding learning mechanism is proposed, which fuses the global coarse-grained and local fine-grained embeddings by multihead attention mechanism for inter-modal and intra-modal contrastive learning, so as to enhance the cross-modal unified representations with stronger discriminativeness and semantic consistency preserving power. With the joint training of intra-modal and inter-modal contrast, the invariant and modal-specific information of different modalities can be maintained in the final unified cross-modal unified hash space. Extensive experiments on several cross-modal benchmark datasets demonstrate that the proposed CMGCH outperforms the state-of the-art methods.

NeurIPS Conference 2024 Conference Paper

SLTrain: a sparse plus low rank approach for parameter and memory efficient pretraining

• Andi Han
• Jiaxiang Li
• Wei Huang
• Mingyi Hong
• Akiko Takeda
• Pratik Jawanpuria
• Bamdev Mishra

Large language models (LLMs) have shown impressive capabilities across various tasks. However, training LLMs from scratch requires significant computational power and extensive memory capacity. Recent studies have explored low-rank structures on weights for efficient fine-tuning in terms of parameters and memory, either through low-rank adaptation or factorization. While effective for fine-tuning, low-rank structures are generally less suitable for pretraining because they restrict parameters to a low-dimensional subspace. In this work, we propose to parameterize the weights as a sum of low-rank and sparse matrices for pretraining, which we call SLTrain. The low-rank component is learned via matrix factorization, while for the sparse component, we employ a simple strategy of uniformly selecting the sparsity support at random and learning only the non-zero entries with the fixed support. While being simple, the random fixed-support sparse learning strategy significantly enhances pretraining when combined with low-rank learning. Our results show that SLTrain adds minimal extra parameters and memory costs compared to pretraining with low-rank parameterization, yet achieves substantially better performance, which is comparable to full-rank training. Remarkably, when combined with quantization and per-layer updates, SLTrain can reduce memory requirements by up to 73% when pretraining the LLaMA 7B model.

NeurIPS Conference 2024 Conference Paper

Unveil Benign Overfitting for Transformer in Vision: Training Dynamics, Convergence, and Generalization

• Jiarui Jiang
• Wei Huang
• Miao Zhang
• Taiji Suzuki
• Liqiang Nie

Transformers have demonstrated great power in the recent development of large foundational models. In particular, the Vision Transformer (ViT) has brought revolutionary changes to the field of vision, achieving significant accomplishments on the experimental side. However, their theoretical capabilities, particularly in terms of generalization when trained to overfit training data, are still not fully understood. To address this gap, this work delves deeply into the \textit{benign overfitting} perspective of transformers in vision. To this end, we study the optimization of a Transformer composed of a self-attention layer with softmax followed by a fully connected layer under gradient descent on a certain data distribution model. By developing techniques that address the challenges posed by softmax and the interdependent nature of multiple weights in transformer optimization, we successfully characterized the training dynamics and achieved generalization in post-training. Our results establish a sharp condition that can distinguish between the small test error phase and the large test error regime, based on the signal-to-noise ratio in the data model. The theoretical results are further verified by experimental simulation. To the best of our knowledge, this is the first work to characterize benign overfitting for Transformers.

TMLR Journal 2023 Journal Article

Analyzing Deep PAC-Bayesian Learning with Neural Tangent Kernel: Convergence, Analytic Generalization Bound, and Efficient Hyperparameter Selection

• Wei Huang
• Chunrui Liu
• Yilan Chen
• Richard Yi Da Xu
• Miao Zhang
• Tsui-Wei Weng

PAC-Bayes is a well-established framework for analyzing generalization performance in machine learning models. This framework provides a bound on the expected population error by considering the sum of training error and the divergence between posterior and prior distributions. In addition to being a successful generalization bound analysis tool, the PAC-Bayesian bound can also be incorporated into an objective function for training probabilistic neural networks, which we refer to simply as {\it Deep PAC-Bayesian Learning}. Deep PAC-Bayesian learning has been shown to achieve competitive expected test set error and provide a tight generalization bound in practice at the same time through gradient descent training. Despite its empirical success, theoretical analysis of deep PAC-Bayesian learning for neural networks is rarely explored. To this end, this paper proposes a theoretical convergence and generalization analysis for Deep PAC-Bayesian learning. For a deep and wide probabilistic neural network, our analysis shows that PAC-Bayesian learning corresponds to solving a kernel ridge regression when the probabilistic neural tangent kernel (PNTK) is used as the kernel. We utilize this outcome in conjunction with the PAC-Bayes $\mathcal{C}$-bound, enabling us to derive an analytical and guaranteed PAC-Bayesian generalization bound for the first time. Finally, drawing insight from our theoretical results, we propose a proxy measure for efficient hyperparameter selection, which is proven to be time-saving on various benchmarks. Our work not only provides a better understanding of the theoretical underpinnings of Deep PAC-Bayesian learning, but also offers practical tools for improving the training and generalization performance of these models.

NeurIPS Conference 2023 Conference Paper

Analyzing Generalization of Neural Networks through Loss Path Kernels

• Yilan Chen
• Wei Huang
• Hao Wang
• Charlotte Loh
• Akash Srivastava
• Lam Nguyen
• Lily Weng

Deep neural networks have been increasingly used in real-world applications, making it critical to ensure their ability to adapt to new, unseen data. In this paper, we study the generalization capability of neural networks trained with (stochastic) gradient flow. We establish a new connection between the loss dynamics of gradient flow and general kernel machines by proposing a new kernel, called loss path kernel. This kernel measures the similarity between two data points by evaluating the agreement between loss gradients along the path determined by the gradient flow. Based on this connection, we derive a new generalization upper bound that applies to general neural network architectures. This new bound is tight and strongly correlated with the true generalization error. We apply our results to guide the design of neural architecture search (NAS) and demonstrate favorable performance compared with state-of-the-art NAS algorithms through numerical experiments.

JBHI Journal 2023 Journal Article

Contrastive Learning for Prediction of Alzheimer's Disease Using Brain 18F-FDG PET

• Yonglin Chen
• Huabin Wang
• Gong Zhang
• Xiao Liu
• Wei Huang
• Xianjun Han
• Xuejun Li
• Melanie Martin

Brain 18F-FDG PET images are commonly-known materials for effectively predicting Alzheimer's disease (AD). However, the data volume of PET is usually insufficient, which is unfavorable to train an accurate AD prediction networks. Furthermore, the PET image is noisy with low signal-to-noise ratio, and simultaneously the feature (metabolic abnormality) used for predicting AD in PET image is not always obvious. Therefore, a contrastive-based learning method is proposed to address the challenges of PET image inherently possessed. Firstly, the slices of 3D PET image are amplified by cropping the image of anchors (i. e. , an augmented version of the same image) to generate extended training data. Meanwhile, contrastive loss is adopted to enlarge inter-class feature distances and reduce intra-class feature differences using subject fuzzy labels as supervised information. Secondly, we construct a double convolutional hybrid attention module to enhance the network to learn different perceptual domains where two convolutional layers with different convolutional kernels ( $7\times 7$ and $5\times 5$ ) are constructed. Moreover, we recommend a diagnosis mechanism by analyzing the consistency of predicted result for PET slices alone with clinical neuropsychological assessment to achieve a better AD diagnosis. The experimental results show that the proposed method outperforms the state-of-the-arts for brain 18F-FDG PET images, and hence demonstrate the advantage of the method in effectively predicting AD.

EAAI Journal 2023 Journal Article

Efficient thermal infrared tracking with cross-modal compress distillation

• Hangfei Li
• Yufei Zha
• Huanyu Li
• Peng Zhang
• Wei Huang

The key issue of thermal infrared tracking is to use neural networks to represent the target effectively and efficiently in the thermal infrared domain. The lack of thermal infrared trainable datasets makes it difficult to train a robust infrared object tracker from scratch, and the time-consuming convolution operations also make the tracking slow. To address the above problems, we proposed cross-modal compression distillation to represent thermal infrared objects for tracking, by leveraging an off-the-shelf RGB model with knowledge distillation. Specifically, cross-modal distillation is performed to effectively transfer knowledge from RGB modality to thermal infrared modality by inputting paired RGB and thermal infrared images into two branches of a Siamese network. Additionally, based on the teacher–student model architecture, the feature extractor is compressed into a lightweight model by model pruning and multi-level deep feature matching. Experimental results on LSOTB-TIR and PTB-TIR datasets show that the thermal infrared object tracking models distilled by our proposed method achieved faster tracking speed with better performance than the baseline RGB tracker by gaining an improvement of 1. 5% Success Rate, 2. 2% Precision, and 1. 9% Normalized Precision, 58 frames per second (FPS) on LSOTB-TIR dataset, respectively.

NeurIPS Conference 2023 Conference Paper

Fed-CO$_{2}$: Cooperation of Online and Offline Models for Severe Data Heterogeneity in Federated Learning

• Zhongyi Cai
• Ye Shi
• Wei Huang
• Jingya Wang

Federated Learning (FL) has emerged as a promising distributed learning paradigm that enables multiple clients to learn a global model collaboratively without sharing their private data. However, the effectiveness of FL is highly dependent on the quality of the data that is being used for training. In particular, data heterogeneity issues, such as label distribution skew and feature skew, can significantly impact the performance of FL. Previous studies in FL have primarily focused on addressing label distribution skew data heterogeneity, while only a few recent works have made initial progress in tackling feature skew issues. Notably, these two forms of data heterogeneity have been studied separately and have not been well explored within a unified FL framework. To address this gap, we propose Fed-CO$_2$, a universal FL framework that handles both label distribution skew and feature skew within a Cooperation mechanism between the Online and Offline models. Specifically, the online model learns general knowledge that is shared among all clients, while the offline model is trained locally to learn the specialized knowledge of each individual client. To further enhance model cooperation in the presence of feature shifts, we design an intra-client knowledge transfer mechanism that reinforces mutual learning between the online and offline models, and an inter-client knowledge transfer mechanism to increase the models’ domain generalization ability. Extensive experiments show that our Fed-CO$_2$ outperforms a wide range of existing personalized federated learning algorithms in terms of handling label distribution skew and feature skew, both individually and collectively. The empirical results are supported by our convergence analyses in a simplified setting.

NeurIPS Conference 2023 Conference Paper

Generator Identification for Linear SDEs with Additive and Multiplicative Noise

• Yuanyuan Wang
• Xi Geng
• Wei Huang
• Biwei Huang
• Mingming Gong

In this paper, we present conditions for identifying the generator of a linear stochastic differential equation (SDE) from the distribution of its solution process with a given fixed initial state. These identifiability conditions are crucial in causal inference using linear SDEs as they enable the identification of the post-intervention distributions from its observational distribution. Specifically, we derive a sufficient and necessary condition for identifying the generator of linear SDEs with additive noise, as well as a sufficient condition for identifying the generator of linear SDEs with multiplicative noise. We show that the conditions derived for both types of SDEs are generic. Moreover, we offer geometric interpretations of the derived identifiability conditions to enhance their understanding. To validate our theoretical results, we perform a series of simulations, which support and substantiate the established findings.

ICLR Conference 2023 Conference Paper

HiViT: A Simpler and More Efficient Design of Hierarchical Vision Transformer

• Xiaosong Zhang 0004
• Yunjie Tian
• Lingxi Xie
• Wei Huang
• Qi Dai 0001
• Qixiang Ye
• Qi Tian 0001

There has been a debate on the choice of plain vs. hierarchical vision transformers, where researchers often believe that the former (e.g., ViT) has a simpler design but the latter (e.g., Swin) enjoys higher recognition accuracy. Recently, the emerge of masked image modeling (MIM), a self-supervised visual pre-training method, raised a new challenge to vision transformers in terms of flexibility, i.e., part of image patches or tokens are to be discarded, which seems to claim the advantages of plain vision transformers. In this paper, we delve deep into the comparison between ViT and Swin, revealing that (i) the performance gain of Swin is mainly brought by a deepened backbone and relative positional encoding, (ii) the hierarchical design of Swin can be simplified into hierarchical patch embedding (proposed in this work), and (iii) other designs such as shifted-window attentions can be removed. By removing the unnecessary operations, we come up with a new architecture named HiViT (short for hierarchical ViT), which is simpler and more efficient than Swin yet further improves its performance on fully-supervised and self-supervised visual representation learning. In particular, after pre-trained using masked autoencoder (MAE) on ImageNet-1K, HiViT-B reports a 84.6% accuracy on ImageNet-1K classification, a 53.3% box AP on COCO detection, and a 52.8% mIoU on ADE20K segmentation, significantly surpassing the baseline. Code is available at https://github.com/zhangxiaosong18/hivit.

IJCAI Conference 2023 Conference Paper

Privacy-Preserving End-to-End Spoken Language Understanding

• Yinggui Wang
• Wei Huang
• Le Yang

Spoken language understanding (SLU), one of the key enabling technologies for human-computer interaction in IoT devices, provides an easy-to-use user interface. Human speech can contain a lot of user-sensitive information, such as gender, identity, and sensitive content. New types of security and privacy breaches have thus emerged. Users do not want to expose their personal sensitive information to malicious attacks by untrusted third parties. Thus, the SLU system needs to ensure that a potential malicious attacker cannot deduce the sensitive attributes of the users, while it should avoid greatly compromising the SLU accuracy. To address the above challenge, this paper proposes a novel SLU multi-task privacy-preserving model to prevent both the speech recognition (ASR) and identity recognition (IR) attacks. The model uses the hidden layer separation technique so that SLU information is distributed only in a specific portion of the hidden layer, and the other two types of information are removed to obtain a privacy-secure hidden layer. In order to achieve good balance between efficiency and privacy, we introduce a new mechanism of model pre-training, namely joint adversarial training, to further enhance the user privacy. Experiments over two SLU datasets show that the proposed method can reduce the accuracy of both the ASR and IR attacks close to that of a random guess, while leaving the SLU performance largely unaffected.

IJCAI Conference 2023 Conference Paper

Self-Supervised Neuron Segmentation with Multi-Agent Reinforcement Learning

• Yinda Chen
• Wei Huang
• Shenglong Zhou
• Qi Chen
• Zhiwei Xiong

The performance of existing supervised neuron segmentation methods is highly dependent on the number of accurate annotations, especially when applied to large scale electron microscopy (EM) data. By extracting semantic information from unlabeled data, self-supervised methods can improve the performance of downstream tasks, among which the mask image model (MIM) has been widely used due to its simplicity and effectiveness in recovering original information from masked images. However, due to the high degree of structural locality in EM images, as well as the existence of considerable noise, many voxels contain little discriminative information, making MIM pretraining inefficient on the neuron segmentation task. To overcome this challenge, we propose a decision-based MIM that utilizes reinforcement learning (RL) to automatically search for optimal image masking ratio and masking strategy. Due to the vast exploration space, using single-agent RL for voxel prediction is impractical. Therefore, we treat each input patch as an agent with a shared behavior policy, allowing for multi-agent collaboration. Furthermore, this multi-agent model can capture dependencies between voxels, which is beneficial for the downstream segmentation task. Experiments conducted on representative EM datasets demonstrate that our approach has a significant advantage over alternative self-supervised methods on the task of neuron segmentation. Code is available at https: //github. com/ydchen0806/dbMiM.

TMLR Journal 2023 Journal Article

Single-Pass Contrastive Learning Can Work for Both Homophilic and Heterophilic Graph

• Haonan Wang
• Jieyu Zhang
• Qi Zhu
• Wei Huang
• Kenji Kawaguchi
• Xiaokui Xiao

Existing graph contrastive learning (GCL) techniques typically require two forward passes for a single instance to construct the contrastive loss, which is effective for capturing the low-frequency signals of node features. Such a dual-pass design has shown empirical success on homophilic graphs, but its effectiveness on heterophilic graphs, where directly connected nodes typically have different labels, is unknown. In addition, existing GCL approaches fail to provide strong performance guarantees. Coupled with the unpredictability of GCL approaches on heterophilic graphs, their applicability in real-world contexts is limited. Then, a natural question arises: Can we design a GCL method that works for both homophilic and heterophilic graphs with a performance guarantee? To answer this question, we theoretically study the concentration property of features obtained by neighborhood aggregation on homophilic and heterophilic graphs, introduce the single-pass graph contrastive learning loss based on the property, and provide performance guarantees for the minimizer of the loss on downstream tasks. As a direct consequence of our analysis, we implement the Single-Pass Graph Contrastive Learning method (SP-GCL). Empirically, on 14 benchmark datasets with varying degrees of homophily, the features learned by the SP-GCL can match or outperform existing strong baselines with significantly less computational overhead, which demonstrates the usefulness of our findings in real-world cases.

NeurIPS Conference 2023 Conference Paper

Understanding and Improving Feature Learning for Out-of-Distribution Generalization

• Yongqiang Chen
• Wei Huang
• Kaiwen Zhou
• Yatao Bian
• Bo Han
• James Cheng

A common explanation for the failure of out-of-distribution (OOD) generalization is that the model trained with empirical risk minimization (ERM) learns spurious features instead of invariant features. However, several recent studies challenged this explanation and found that deep networks may have already learned sufficiently good features for OOD generalization. Despite the contradictions at first glance, we theoretically show that ERM essentially learns both spurious and invariant features, while ERM tends to learn spurious features faster if the spurious correlation is stronger. Moreover, when fed the ERM learned features to the OOD objectives, the invariant feature learning quality significantly affects the final OOD performance, as OOD objectives rarely learn new features. Therefore, ERM feature learning can be a bottleneck to OOD generalization. To alleviate the reliance, we propose Feature Augmented Training (FeAT), to enforce the model to learn richer features ready for OOD generalization. FeAT iteratively augments the model to learn new features while retaining the already learned features. In each round, the retention and augmentation operations are performed on different subsets of the training data that capture distinct features. Extensive experiments show that FeAT effectively learns richer features thus boosting the performance of various OOD objectives.

IJCAI Conference 2022 Conference Paper

Biological Instance Segmentation with a Superpixel-Guided Graph

• Xiaoyu Liu
• Wei Huang
• Yueyi Zhang
• Zhiwei Xiong

Recent advanced proposal-free instance segmentation methods have made significant progress in biological images. However, existing methods are vulnerable to local imaging artifacts and similar object appearances, resulting in over-merge and over-segmentation. To reduce these two kinds of errors, we propose a new biological instance segmentation framework based on a superpixel-guided graph, which consists of two stages, i. e. , superpixel-guided graph construction and superpixel agglomeration. Specifically, the first stage generates enough superpixels as graph nodes to avoid over-merge, and extracts node and edge features to construct an initialized graph. The second stage agglomerates superpixels into instances based on the relationship of graph nodes predicted by a graph neural network (GNN). To solve over-segmentation and prevent introducing additional over-merge, we specially design two loss functions to supervise the GNN, i. e. , a repulsion-attraction (RA) loss to better distinguish the relationship of nodes in the feature space, and a maximin agglomeration score (MAS) loss to pay more attention to crucial edge classification. Extensive experiments on three representative biological datasets demonstrate the superiority of our method over existing state-of-the-art methods. Code is available at https: //github. com/liuxy1103/BISSG.

NeurIPS Conference 2022 Conference Paper

Deep Active Learning by Leveraging Training Dynamics

• Haonan Wang
• Wei Huang
• Ziwei Wu
• Hanghang Tong
• Andrew J Margenot
• Jingrui He

Active learning theories and methods have been extensively studied in classical statistical learning settings. However, deep active learning, i. e. , active learning with deep learning models, is usually based on empirical criteria without solid theoretical justification, thus suffering from heavy doubts when some of those fail to provide benefits in applications. In this paper, by exploring the connection between the generalization performance and the training dynamics, we propose a theory-driven deep active learning method (dynamicAL) which selects samples to maximize training dynamics. In particular, we prove that the convergence speed of training and the generalization performance is positively correlated under the ultra-wide condition and show that maximizing the training dynamics leads to a better generalization performance. Furthermore, to scale up to large deep neural networks and data sets, we introduce two relaxations for the subset selection problem and reduce the time complexity from polynomial to constant. Empirical results show that dynamicAL not only outperforms the other baselines consistently but also scales well on large deep learning models. We hope our work inspires more attempts in bridging the theoretical findings of deep networks and practical impacts in deep active learning applications.

NeurIPS Conference 2022 Conference Paper

Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis

• Wuyang Chen
• Wei Huang
• Xinyu Gong
• Boris Hanin
• Zhangyang Wang

Advanced deep neural networks (DNNs), designed by either human or AutoML algorithms, are growing increasingly complex. Diverse operations are connected by complicated connectivity patterns, e. g. , various types of skip connections. Those topological compositions are empirically effective and observed to smooth the loss landscape and facilitate the gradient flow in general. However, it remains elusive to derive any principled understanding of their effects on the DNN capacity or trainability, and to understand why or in which aspect one specific connectivity pattern is better than another. In this work, we theoretically characterize the impact of connectivity patterns on the convergence of DNNs under gradient descent training in fine granularity. By analyzing a wide network's Neural Network Gaussian Process (NNGP), we are able to depict how the spectrum of an NNGP kernel propagates through a particular connectivity pattern, and how that affects the bound of convergence rates. As one practical implication of our results, we show that by a simple filtration of "unpromising" connectivity patterns, we can trim down the number of models to evaluate, and significantly accelerate the large-scale neural architecture search without any overhead.

NeurIPS Conference 2022 Conference Paper

Interpreting Operation Selection in Differentiable Architecture Search: A Perspective from Influence-Directed Explanations

• Miao Zhang
• Wei Huang
• Bin Yang

The Differentiable ARchiTecture Search (DARTS) has dominated the neural architecture search community due to its search efficiency and simplicity. DARTS leverages continuous relaxation to convert the intractable operation selection problem into a continuous magnitude optimization problem which can be easily handled with gradient-descent, while it poses an additional challenge in measuring the operation importance or selecting an architecture from the optimized magnitudes. The vanilla DARTS assumes the optimized magnitudes reflect the importance of operations, while more recent works find this naive assumption leads to poor generalization and is without any theoretical guarantees. In this work, we leverage influence functions, the functional derivatives of the loss function, to theoretically reveal the operation selection part in DARTS and estimate the candidate operation importance by approximating its influence on the supernet with Taylor expansions. We show the operation strength is not only related to the magnitude but also second-order information, leading to a fundamentally new criterion for operation selection in DARTS, named Influential Magnitude. Empirical studies across different tasks on several spaces show that vanilla DARTS and its variants can avoid most failures by leveraging the proposed theory-driven operation selection criterion.

AAAI Conference 2022 Conference Paper

Learning to Model Pixel-Embedded Affinity for Homogeneous Instance Segmentation

• Wei Huang
• Shiyu Deng
• Chang Chen
• Xueyang Fu
• Zhiwei Xiong

Homogeneous instance segmentation aims to identify each instance in an image where all interested instances belong to the same category, such as plant leaves and microscopic cells. Recently, proposal-free methods, which straightforwardly generate instance-aware information to group pixels into different instances, have received increasing attention due to their efficient pipeline. However, they often fail to distinguish adjacent instances due to similar appearances, dense distribution and ambiguous boundaries of instances in homogeneous images. In this paper, we propose a pixel-embedded affinity modeling method for homogeneous instance segmentation, which is able to preserve the semantic information of instances and improve the distinguishability of adjacent instances. Instead of predicting affinity directly, we propose a self-correlation module to explicitly model the pairwise relationships between pixels, by estimating the similarity between embeddings generated from the input image through CNNs. Based on the self-correlation module, we further design a cross-correlation module to maintain the semantic consistency between instances. Specifically, we map the transformed input images with different views and appearances into the same embedding space, and then mutually estimate the pairwise relationships of embeddings generated from the original input and its transformed variants. In addition, to integrate the global instance information, we introduce an embedding pyramid module to model affinity on different scales. Extensive experiments demonstrate the versatile and superior performance of our method on three representative datasets. Code and models are available at https: //github. com/weih527/ Pixel-Embedded-Affinity.

NeurIPS Conference 2022 Conference Paper

MissDAG: Causal Discovery in the Presence of Missing Data with Continuous Additive Noise Models

• Erdun Gao
• Ignavier Ng
• Mingming Gong
• Li Shen
• Wei Huang
• Tongliang Liu
• Kun Zhang
• Howard Bondell

State-of-the-art causal discovery methods usually assume that the observational data is complete. However, the missing data problem is pervasive in many practical scenarios such as clinical trials, economics, and biology. One straightforward way to address the missing data problem is first to impute the data using off-the-shelf imputation methods and then apply existing causal discovery methods. However, such a two-step method may suffer from suboptimality, as the imputation algorithm may introduce bias for modeling the underlying data distribution. In this paper, we develop a general method, which we call MissDAG, to perform causal discovery from data with incomplete observations. Focusing mainly on the assumptions of ignorable missingness and the identifiable additive noise models (ANMs), MissDAG maximizes the expected likelihood of the visible part of observations under the expectation-maximization (EM) framework. In the E-step, in cases where computing the posterior distributions of parameters in closed-form is not feasible, Monte Carlo EM is leveraged to approximate the likelihood. In the M-step, MissDAG leverages the density transformation to model the noise distributions with simpler and specific formulations by virtue of the ANMs and uses a likelihood-based causal discovery algorithm with directed acyclic graph constraint. We demonstrate the flexibility of MissDAG for incorporating various causal discovery algorithms and its efficacy through extensive simulations and real data experiments.

JBHI Journal 2022 Journal Article

Network Theory Based EHG Signal Analysis and its Application in Preterm Prediction

• Jinshan Xu
• Mengting Wang
• Jinpeng Zhang
• Zhenqin Chen
• Wei Huang
• Guojiang Shen
• Meiyu Zhang

Objective: Preterm birth is the leading cause of neonatal morbidity and mortality. Early identification of high-risk patients followed by medical interventions is essential to the prevention of preterm birth. Based on the relationship between uterine contraction and the fundamental electrical activities of muscles, we extracted effective features from EHG signals recorded from pregnant women, and use them to train classifiers with the purpose of providing high precision in classifying term and preterm pregnancies. Methods: To characterize changes from irregularity to coherence of the uterine activity during the whole pregnancy, network representations of the original electrohysterogram (EHG) signals are established by applying the Horizontal Visibility Graph (HVG) algorithm, from which we extract network degree density and distribution, clustering coefficient and assortativity coefficient. Concerns on the interferences of different noise sources embedded in the EHG signal, we apply Short-Time Fourier Transform (STFT) to expand the original signal in the time-frequency domain. This allows a network representation and the extraction of related features on each frequency component. Feature selection algorithms are then used to filter out unrelated frequency components. We further apply the proposed feature extraction method to EHG signals available in the Term-Preterm EHG database (TPEHG), and use them to train classifiers. We adopt the Partition-Synthesis scheme which splits the original imbalanced dataset into two sets, and synthesizes artificial samples separately within each subset to solve the problem of dataset imbalance. Results: The optimally selected network-based features, not only contribute to the identification of the essential frequency components of uterine activities related to preterm birth, but also to improved performance in classifying term/preterm pregnancies, i. e. , the SVM (Support Vector Machine) classifier trained with the available samples in the TPEHG gives sensitivity, specificity, overall accuracy, and $auc$ values as high as 0. 89, 0. 93, 0. 91, and 0. 97, respectively.

NeurIPS Conference 2022 Conference Paper

Weighted Mutual Learning with Diversity-Driven Model Compression

• Miao Zhang
• Li Wang
• David Campos
• Wei Huang
• Chenjuan Guo
• Bin Yang

Online distillation attracts attention from the community as it simplifies the traditional two-stage knowledge distillation process into a single stage. Online distillation collaboratively trains a group of peer models, which are treated as students, and all students gain extra knowledge from each other. However, memory consumption and diversity among peers are two key challenges to the scalability and quality of online distillation. To address the two challenges, this paper presents a framework called Weighted Mutual Learning with Diversity-Driven Model Compression (WML) for online distillation. First, at the base of a hierarchical structure where peers share different parts, we leverage the structured network pruning to generate diversified peer models and reduce the memory requirements. Second, rather than taking the average of peers, this paper, for the first time, leverages a bi-level formulation to estimate the relative importance of peers with a close-form, to further boost the effectiveness of the distillation from each other. Extensive experiments show the generalization of the proposed framework, which outperforms existing online distillation methods on a variety of deep neural networks. More interesting, as a byproduct, \WML produces a series of pruned models under different model sizes in a single run, which also achieves competitive results compared with existing channel pruning methods.

IJCAI Conference 2021 Conference Paper

Item Response Ranking for Cognitive Diagnosis

• Shiwei Tong
• Qi Liu
• Runlong Yu
• Wei Huang
• Zhenya Huang
• Zachary A. Pardos
• Weijie Jiang

Cognitive diagnosis, a fundamental task in education area, aims at providing an approach to reveal the proficiency level of students on knowledge concepts. Actually, monotonicity is one of the basic conditions in cognitive diagnosis theory, which assumes that student's proficiency is monotonic with the probability of giving the right response to a test item. However, few of previous methods consider the monotonicity during optimization. To this end, we propose Item Response Ranking framework (IRR), aiming at introducing pairwise learning into cognitive diagnosis to well model the monotonicity between item responses. Specifically, we first use an item specific sampling method to sample item responses and construct response pairs based on their partial order, where we propose the two-branch sampling methods to handle the unobserved responses. After that, we use a pairwise objective function to exploit the monotonicity in the pair formulation. In fact, IRR is a general framework which can be applied to most of contemporary cognitive diagnosis models. Extensive experiments demonstrate the effectiveness and interpretability of our method.

NeurIPS Conference 2021 Conference Paper

On the Equivalence between Neural Network and Support Vector Machine

• Yilan Chen
• Wei Huang
• Lam Nguyen
• Tsui-Wei Weng

Recent research shows that the dynamics of an infinitely wide neural network (NN) trained by gradient descent can be characterized by Neural Tangent Kernel (NTK) \citep{jacot2018neural}. Under the squared loss, the infinite-width NN trained by gradient descent with an infinitely small learning rate is equivalent to kernel regression with NTK \citep{arora2019exact}. However, the equivalence is only known for ridge regression currently \citep{arora2019harnessing}, while the equivalence between NN and other kernel machines (KMs), e. g. support vector machine (SVM), remains unknown. Therefore, in this work, we propose to establish the equivalence between NN and SVM, and specifically, the infinitely wide NN trained by soft margin loss and the standard soft margin SVM with NTK trained by subgradient descent. Our main theoretical results include establishing the equivalence between NN and a broad family of $\ell_2$ regularized KMs with finite-width bounds, which cannot be handled by prior work, and showing that every finite-width NN trained by such regularized loss functions is approximately a KM. Furthermore, we demonstrate our theory can enable three practical applications, including (i) \textit{non-vacuous} generalization bound of NN via the corresponding KM; (ii) \textit{nontrivial} robustness certificate for the infinite-width NN (while existing robustness verification methods would provide vacuous bounds); (iii) intrinsically more robust infinite-width NNs than those from previous kernel regression.

IJCAI Conference 2021 Conference Paper

On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization

• Wei Huang
• Weitao Du
• Richard Yi Da Xu

The prevailing thinking is that orthogonal weights are crucial to enforcing dynamical isometry and speeding up training. The increase in learning speed that results from orthogonal initialization in linear networks has been well-proven. However, while the same is believed to also hold for nonlinear networks when the dynamical isometry condition is satisfied, the training dynamics behind this contention have not been thoroughly explored. In this work, we study the dynamics of ultra-wide networks across a range of architectures, including Fully Connected Networks (FCNs) and Convolutional Neural Networks (CNNs) with orthogonal initialization via neural tangent kernel (NTK). Through a series of propositions and lemmas, we prove that two NTKs, one corresponding to Gaussian weights and one to orthogonal weights, are equal when the network width is infinite. Further, during training, the NTK of an orthogonally-initialized infinite-width network should theoretically remain constant. This suggests that the orthogonal initialization cannot speed up training in the NTK (lazy training) regime, contrary to the prevailing thoughts. In order to explore under what circumstances can orthogonality accelerate training, we conduct a thorough empirical investigation outside the NTK regime. We find that when the hyper-parameters are set to achieve a linear regime in nonlinear activation, orthogonal initialization can improve the learning speed with a large learning rate or large depth.

JMLR Journal 2021 Journal Article

Transferability of Spectral Graph Convolutional Neural Networks

• Ron Levie
• Wei Huang
• Lorenzo Bucci
• Michael Bronstein
• Gitta Kutyniok

This paper focuses on spectral graph convolutional neural networks (ConvNets), where filters are defined as elementwise multiplication in the frequency domain of a graph. In machine learning settings where the data set consists of signals defined on many different graphs, the trained ConvNet should generalize to signals on graphs unseen in the training set. It is thus important to transfer ConvNets between graphs. Transferability, which is a certain type of generalization capability, can be loosely defined as follows: if two graphs describe the same phenomenon, then a single filter or ConvNet should have similar repercussions on both graphs. This paper aims at debunking the common misconception that spectral filters are not transferable. We show that if two graphs discretize the same “continuous” space, then a spectral filter or ConvNet has approximately the same repercussion on both graphs. Our analysis is more permissive than the standard analysis. Transferability is typically described as the robustness of the filter to small graph perturbations and re-indexing of the vertices. Our analysis accounts also for large graph perturbations. We prove transferability between graphs that can have completely different dimensions and topologies, only requiring that both graphs discretize the same underlying space in some generic sense. [abs] [ pdf ][ bib ] &copy JMLR 2021. ( edit, beta )

AAAI Conference 2019 Conference Paper

A Framework to Coordinate Segmentation and Recognition

• Wei Huang
• Huimin Yu
• Weiwei Zheng
• Jing Zhang

A novel coordination framework between the segmentation and the recognition is proposed, to conduct the two tasks collaboratively and iteratively. To accomplish the cooperation, objects are expressed in two aspects: shape and appearance, which are learned and leveraged as constraints to the segmentation so that the object segmentation mask will be consistent with the object regions in the image and the knowledge we have. For the shape, a bottom-top-bottom pathway is built using an encoder-decoder network with capsule neurons, where the encoder extracts the features of the shape that used for recognition and the decoder generates reference shapes according to these features and the recognition result. During this procedure, capsule neurons can parse the existence of the object and cope with the interference in the segmentation. The appearance knowledge is utilized in another pathway to assist the segmentation processing. Both the shape and appearance information are dependent on the recognition result, thus allowing the classifier to convey object information to the segmenter. Experiments demonstrate the effectiveness of our framework and model in collaboratively segmenting and recognizing objects that can be recognized using their shapes/shape-patterns.

IJCAI Conference 2009 Conference Paper

• Wei Huang
• Zhonghua Wen
• YUNFEI JIANG
• Hong Peng

In many real world planning domains, some observation information is optional and useless to the execution of a plan; on the other hand, information acquisition may require some kind of cost. The problem of observation reduction for strong plans has been addressed in the literature. However, observation reduction for plans with contexts (which are more general and useful than strong plans in robotics) is still a open problem. In this paper, we present an attempt to solve the problem. Our first contribution is the definition of structured plans, which can encode sequential, conditional and iterative behaviors, and is expressive enough for dealing with incomplete observation information and internal states of the agent. A second contribution is an observation reduction algorithm for plans with contexts, which can transform a plan with contexts into a structured plan that only branches on necessary observation information.

IJCAI Conference 2007 Conference Paper

• Wei Huang
• Zhonghua Wen
• YUNFEI JIANG
• Lihua Wu

Strong planning under full or partial observability has been addressed in the literature. But this research line is carried out under the hypothesis that the set of observation variables is fixed and compulsory. In most real world domains, however, observation variables are optional and many of them are useless in the execution of a plan; on the other side, information acquisition may require some kind of cost. So it is significant to find a minimal set of observation variables which are necessary for the execution of a plan, and to best of our knowledge, it is still an open problem. In this paper we present a first attempt to solve the problem, namely, we define an algorithm that finds an approximate minimal set of observation variables which are necessary for the execution of a strong plan under full observability (i. e. a state-action table); and transforms the plan into a strong plan under partial observability (i. e. a conditional plan branching on the observations built on these observation variables).

YNIMG Journal 1996 Journal Article

MRI detection of the murine brain response to light

• Ildiko Palyka
• Wei Huang
• Edward M. Eisenstein
• Nora D. Volkow
• Charles S. Springer