Yang Zhang

JBHI Journal 2026 Journal Article

A Real-Time Scale-Robust Network for Glottis Segmentation in Nasal Transnasal Intubation

• Yang Zhou
• Chaoyong Zhang
• Ruoyi Hao
• Huilin Pan
• Yang Zhang
• Hongliang Ren

Nasotracheal intubation (NTI) is a critical clinical procedure for establishing and maintaining patient airway patency. Machine-assisted NTI has emerged as a pivotal approach for optimizing procedural efficiency and minimizing manual intervention. However, visual detection algorithms employed for NTI navigation encounter significant challenges, including complex anatomical environments and suboptimal illumination conditions surrounding the glottis. Additionally, the glottis presents considerable scale variability throughout the procedure, initially appearing as a small, difficult-to-capture structure before expanding to occupy nearly the entire field of view. Moreover, traditional visual detection methods often have high computational costs, making real-time, high-precision detection on portable devices challenging. To enhance NTI efficacy and address these challenges, this paper proposes a novel glottis segmentation framework optimized for vision-assisted NTI applications. First, we designed a lightweight, multi-receptive field feature extraction module to reduce intra-class differences, achieving robustness to scale variations of the glottis. This module was then stacked to form the backbone and neck of our network. Subsequently, we developed an advanced label assignment method and redefined the number of samples to further reduce intra-class differences and enhance accuracy in the complex NTI environment. Experiments on three distinct datasets demonstrate that our network surpasses state-of-the-art algorithms, achieving a segmentation mDice of 92. 9% with a compact model size of 19 MB and an inference speed exceeding 170 frames per second.

JBHI Journal 2026 Journal Article

BreastUS-Net: An Attention-Guided Dual-Branch Network With Feature Fusion for Fine-Grained Breast Tumor Classification in Ultrasound Imaging

• Sohaib Asif
• Di Ou
• Fazal Hadi
• Yuqi Yan
• Enyu Wang
• Yang Zhang
• Dong Xu

Despite advances in deep learning (DL) and computer vision, breast cancer (BC) detection via ultrasound remains challenging. Existing methods often focus on single tasks using complex pipelines and publicly available datasets, limiting clinical applicability. To address this, we propose BreastUS-Net—a novel architecture for hierarchical BC classification using diverse datasets. Our approach uses a dual-branch MobileNet architecture with fine-tuned and frozen layers to capture both task-specific and general features, eliminating manual feature extraction. These features are then fused to create a comprehensive representation, which is subsequently aggregated and refined. The aggregation step merges the outputs from both branches, while the refinement module reduces complexity, highlights relevant patterns, and mitigates overfitting to improve generalization. Additionally, we integrate a multihead self-attention (MHSA) block to highlight diagnostically significant regions in ultrasound images, enhancing both accuracy and robustness. Finally, the orthogonal softmax layer (OSL) boosts discriminative power by enforcing orthogonality among weight vectors, reducing parameter co-adaptation and enabling more effective optimization. We used six diverse datasets from multiple centers, including: a large Zhejiang Cancer Hospital set (2, 171 images), public BUSI dataset (780 images), external test sets from Yunnan Cancer Hospital (351 images) and Sir Run Run Shaw Hospitals (365 images), fibroadenoma (FA) vs. phyllodes tumor (PT) classification, and a PT grading dataset. We use explainable AI (XAI) techniques—Grad-CAM, SHAP, and saliency maps—to enhance trust in breast ultrasound predictions. Our model achieves state-of-the-art performance, with accuracies of 94. 48% on a clinical dataset and 94. 23% on the BUSI dataset, highlighting its potential to improve BC diagnosis and personalized treatment.

AAAI Conference 2026 Conference Paper

Coordinated Humanoid Robot Locomotion with Symmetry Equivariant Reinforcement Learning Policy

• Buqing Nie
• Yang Zhang
• Rongjun Jin
• Zhanxiang Cao
• Huangxuan Lin
• Xiaokang Yang
• Yue Gao

The human nervous system exhibits bilateral symmetry, enabling coordinated and balanced movements. However, existing Deep Reinforcement Learning (DRL) methods for humanoid robots neglect morphological symmetry of the robot, leading to uncoordinated and suboptimal behaviors. Inspired by human motor control, we propose Symmetry Equivariant Policy (SE-Policy), a new DRL framework that embeds strict symmetry equivariance in the actor and symmetry invariance in the critic without additional hyperparameters. SE-Policy enforces consistent behaviors across symmetric observations, producing temporally and spatially coordinated motions with higher task performance. Extensive experiments on velocity tracking tasks, conducted in both simulation and real-world deployment with the Unitree G1 humanoid robot, demonstrate that SE-Policy improves tracking accuracy by up to 40% compared to state-of-the-art baselines, while achieving superior spatial-temporal coordination. These results demonstrate the effectiveness of SE-Policy and its broad applicability to humanoid robots.

AAAI Conference 2026 Conference Paper

Don’t Start Over: A Cost-Effective Framework for Migrating Personalized Prompts Between LLMs

• Ziyi Zhao
• Chongming Gao
• Yang Zhang
• Haoyan Liu
• Weinan Gan
• Huifeng Guo
• Yong Liu
• Fuli Feng

Personalization in Large Language Models (LLMs) often relies on user-specific soft prompts. However, these prompts become obsolete when the foundation model is upgraded, necessitating costly, full-scale retraining. To overcome this limitation, we propose the Prompt-level User Migration Adapter (PUMA), a lightweight framework to efficiently migrate personalized prompts across incompatible models. PUMA utilizes a parameter-efficient adapter to bridge the semantic gap, combined with a group-based user selection strategy to significantly reduce training costs. Experiments on three large-scale datasets show our method matches or even surpasses the performance of retraining from scratch, reducing computational cost by up to 98%. The framework demonstrates strong generalization across diverse model architectures and robustness in advanced scenarios like chained and aggregated migrations, offering a practical path for the sustainable evolution of personalized AI by decoupling user assets from the underlying models.

EAAI Journal 2026 Journal Article

Hard constraints and soft learning dual-graph anomaly detection for industrial processes

• Chuan Zhang
• Ming-Qing Zhang
• Yi Luo
• Wei Ke
• Qun-Xiong Zhu
• Yan-Lin He
• Yang Zhang
• Yuan Xu

Anomaly detection is critical for safe and stable operation in industrial processes. Industrial data exhibits strong spatiotemporal dependence, while variable interactions often evolve dynamically. Traditional methods struggle to model both fixed physical constraints and dynamic data relationships. This paper proposes a hard constraints and soft learning dual-graph anomaly detection (HCSL-DGAD). First, a macro-graph with hard constraints is constructed based on the connections between various components of the industrial process, and micro-graphs with soft learning is constructed through an adaptive method based on attention mechanisms. The macro-graph transmits physical constraints through a spatiotemporal graph convolutional network to ensure the rationality of the abnormal propagation path. The micro-graph uses a multi-head attention mechanism to dynamically learn the implicit relationship between nodes and capture coupling information not covered by the physical topology. Secondly, to address the multi-scale anomalies in the spatiotemporal domain, a dual-channel architecture is employed to extract features from both the macro and micro-graphs. Edge weights and node states are alternately updated in the micro-graph channel to accurately identify anomaly patterns at different scales. At the same time, temporal attention and variable attention are combined in the macro-graph channel to jointly improve the detection accuracy. Extensive experiments on three benchmarks show that HCSL-DGAD achieves average F1-scores of 98. 24%, 89. 92%, and 87. 00% on the Tennessee Eastman (TE) process, Secure Water Treatment (SWaT) and PROcess NeTwork Optimization (PRONTO) datasets, respectively.

AAAI Conference 2026 Conference Paper

Hyperbolic-Enhanced Mixture-of-Experts Mamba for Sequential Recommendation

• Yuwen Liu
• Lianyong Qi
• Xingyuan Mao
• Weiming Liu
• Xuhui Fan
• Qiang Ni
• Xuyun Zhang
• Yang Zhang

Sequential recommendation has emerged as a fundamental task in various domains, aiming to predict a user's next interaction based on historical behavior. Recent advances in deep sequence models, particularly Transformer-based architectures and the more recent Mamba, have substantially pushed the boundaries of sequential modeling performance. However, existing methods still face two critical challenges. First, many current approaches overlook the hierarchical structures and high-order dependencies among items, typically restricting representation learning to conventional Euclidean spaces, which limits their capacity to capture complex relational information. Second, although Mamba excels at long-range dependency modeling, its reliance on static Feed-Forward Networks (FFNs) hinders its ability to dynamically adapt to evolving user preferences across diverse contexts. To address these limitations, we propose a Hyperbolic-Enhanced Mixture-of-Experts Mamba recommender (HM2Rec) for sequential recommendation. HM2Rec first encodes user-item relationships through hyperbolic graph convolution to exploit hierarchical structure more effectively. Then, a Variational Graph Auto-Encoder (VGAE) is employed to reconstruct node embeddings, improving structural robustness. To further enhance sequential modeling, we integrate Rotary Positional Encoding (RoPE) into Mamba to better capture relative position dependencies, and replace the FFN with Mixture-of-Expert (MOE) module, enabling dynamic and personalized expert selection for each token. Our extensive experiments on four widely-used public datasets demonstrate that HM2Rec outperforms several advanced baseline models.

AAAI Conference 2026 Conference Paper

Keep On Going: Learning Robust Humanoid Motion Skills via Selective Adversarial Training

• Yang Zhang
• Zhanxiang Cao
• Buqing Nie
• Haoyang Li
• Zhong Jiangwei
• Qiao Sun
• Xiaoyi Hu
• Xiaokang Yang

Humanoid robots are expected to operate reliably over long horizons while executing versatile whole-body skills. Yet Reinforcement Learning (RL) motion policies typically lose stability under prolonged operation, sensor/actuator noise, and real world disturbances. In this work, we propose a Selective Adversarial Attack for Robust Training (SA2RT) to enhance the robustness of motion skills. The adversary is learned to identify and sparsely perturb the most vulnerable states and actions under an attack-budget constraint, thereby exposing true weakness without inducing conservative overfitting. The resulting non-zero sum, alternating optimization continually strengthens the motion policy against the strongest discovered attacks. We validate our approach on the Unitree G1 humanoid robot across perceptive locomotion and whole-body control tasks. Experimental results show that adversarially trained policies improve the terrain traversal success rate by 40%, reduce the trajectory tracking error by 32%, and maintain long horizon mobility and tracking performance. Together, these results demonstrate that selective adversarial attacks are an effective driver for learning robust, long horizon humanoid motion skills.

AAAI Conference 2026 Conference Paper

Plug-and-Play Parameter-Efficient Tuning of Embeddings for Federated Recommendation

• Haochen Yuan
• Yang Zhang
• Xiang He
• Quan Z. Sheng
• Zhongjie Wang

With the rise of cloud-edge collaboration, recommendation services are increasingly trained in distributed environments. Federated Recommendation (FR) enables such multi-end collaborative training while preserving privacy by sharing model parameters instead of raw data. However, the large number of parameters, primarily due to the massive item embeddings, significantly hampers communication efficiency. While existing studies mainly focus on improving the efficiency of FR models, they largely overlook the issue of embedding parameter overhead. To address this gap, we propose a FR training framework with Parameter-Efficient Fine-Tuning (PEFT) based embedding designed to reduce the volume of embedding parameters that need to be transmitted. Our approach offers a lightweight, plugin-style solution that can be seamlessly integrated into existing FR methods. In addition to incorporating common PEFT techniques such as LoRA and Hash-based encoding, we explore the use of Residual Quantized Variational Autoencoders (RQ-VAE) as a novel PEFT strategy within our framework. Extensive experiments across various FR model backbones and datasets demonstrate that our framework significantly reduces communication overhead while improving accuracy.

AAAI Conference 2026 Conference Paper

Pre-Trained Video Generative Models as World Simulators

• Haoran He
• Yang Zhang
• Liang Lin
• Zhongwen Xu
• Ling Pan

Video generative models pre-trained on large-scale internet datasets have achieved remarkable success, excelling at producing realistic synthetic videos. However, they often generate clips based on static prompts (e.g., text or images), limiting their ability to model interactive and dynamic scenarios. In this paper, we propose Dynamic World Simulation (DWS), a novel approach to transform pre-trained video generative models into controllable world simulators capable of executing specified action trajectories. To achieve precise alignment between conditioned actions and generated visual changes, we introduce a lightweight, universal action-conditioned module that seamlessly integrates into any existing model. Instead of focusing on complex visual details, we demonstrate that consistent dynamic transition modeling is the key to building powerful world simulators. Building upon this insight, we further introduce a motion-reinforced loss that enhances action controllability by compelling the model to capture dynamic changes more effectively. Experiments demonstrate that DWS can be versatilely applied to both diffusion and autoregressive transformer models, achieving significant improvements in generating action-controllable, dynamically consistent videos across games and robotics domains. Moreover, to facilitate the applications of the learned world simulator in downstream tasks such as model-based reinforcement learning, we propose prioritized imagination to improve sample efficiency, demonstrating competitive performance compared with state-of-the-art methods.

EAAI Journal 2026 Journal Article

Pursuit-evasion game of unmanned tracked vehicles based on multi-agent twin delayed deep deterministic policy gradient

• Jiangyi Yao
• Xiongwei Li
• Yang Zhang
• Kaiyan Chen
• Yongbo Li
• Jingyu Ji

The pursuit-evasion (PE) game, fundamental to unmanned swarm operations in complex battlefields, presents significant challenges in scalability and adversarial strategy optimization. This paper investigates the multi-agent PE problem for unmanned tracked vehicles (UTVs) within a Markov game framework. To address the curse of dimensionality, we propose a novel warning area mechanism that compresses and standardizes the perceptual state space into a fixed-length representation, enabling the algorithm to maintain performance as the number of agents and obstacles varies. Furthermore, we formulate the interaction between pursuers and evaders as a zero-sum game, training both sides simultaneously to foster the emergence of robust pursuer strategies against a strong, adaptive adversary. A group nearest-neighbor target selection mechanism is integrated to enhance cooperative capture efficiency. Extensive simulations demonstrate that our method, built upon the multi-agent twin delayed deep deterministic policy gradient (MATD3) algorithm, achieves superior performance in capture success rate and collision avoidance compared to baseline approaches, while exhibiting remarkable generalization capabilities across diverse scenarios. This work provides a scalable and transferable framework for developing advanced decision-making intelligence in collaborative unmanned ground systems.

AAAI Conference 2026 Conference Paper

QiMeng-CRUX: Narrowing the Gap Between Natural Language and Verilog via Core Refined Understanding eXpression

• Lei Huang
• Rui Zhang
• Jiaming Guo
• Yang Zhang
• Di Huang
• Shuyao Cheng
• Pengwei Jin
• Chongxiao Li

Large language models (LLMs) have shown promising capabilities in hardware description language (HDL) generation. However, existing approaches often rely on free-form natural language descriptions that are often ambiguous, redundant, and unstructured, which poses significant challenges for downstream Verilog code generation. We treat hardware code generation as a complex transformation from an open-ended natural language space to a domain-specific, highly constrained target space. To bridge this gap, we introduce Core Refined Understanding eXpression (CRUX), a structured intermediate space that captures the essential semantics of user intent while organizing the expression for precise Verilog code generation. We further design a two-stage training framework, comprising Joint Expression Modeling and Dual-Space Optimization, to enhance the quality of both CRUX and Verilog code. Experiments across multiple Verilog generation benchmarks demonstrate that our model, QiMeng-CRUX, achieves state-of-the-art performance among general models, particularly under challenging design tasks. Furthermore, the CRUX space proves transferable and beneficial when used as input prompts for other code models, highlighting its effectiveness in narrowing the gap between free-form natural language descriptions and precise Verilog generation.

EAAI Journal 2026 Journal Article

Retraction notice to “A novel integrated price and load forecasting method in smart grid environment based on multi-level structure” [Eng. Appl. Artif. Intell. 95 (2020) 103852]

• Yang Zhang
• Caibo Deng
• Ran Zhao
• Sebastian leto

AAAI Conference 2026 Conference Paper

Right Branches Matter in Failure-based Variable Ordering Heuristics

• Yang Zhang
• Hongbo Li

Failure-based variable ordering heuristics (VOH) are efficient general-purpose search heuristics for solving constraint satisfaction problems (CSP). They learn from the failures detected during the search and select the variables that are most likely to fail. The current failure-based VOHs, i.e. the failure-rate-based (FRBA) and failure-length-based (FLBA), focus on only the failures detected in left branches. In this paper, we investigate how the failure information from right branches affects the performance of the failure-based VOHs. Four strategies utilizing the failure information of right branches are proposed to refine the failure-based VOHs. Our experiments performed with the benchmark instances used in the recent MiniZinc challenges show that utilizing the failures detected in right branches enhances the performance of the failure-based VOHs. The refined version combining all the proposed strategies generally gets the best performance. It demonstrates remarkable superiority over several general-purpose VOHs, including activity-based search, conflict-history search, refined weighted degree, pick/dom, and the existing FRBA, which are considered state-of-the-art. Our study demonstrates that right branches matter in failure-based VOHs.

AAAI Conference 2026 Conference Paper

SAME: Spatial-Aware Multimodal Egocentric Human Pose Estimation

• Yurong Fu
• Peng Dai
• Yu Zhang
• Feng Yiqiang
• Yang Zhang
• Haoqian Wang

Egocentric human pose estimation (HPE) plays a crucial role in immersive applications such as virtual and augmented reality. However, existing methods relying on either visual or sparse inertial data alone often suffer from occlusion or ill-posed problems. In this work, we propose SAME, a novel spatial-aware multimodal fusion framework combining the complementary signals from the stereo images and sparse IMUs for accurate and robust egocentric HPE. It adopts a two-stage network based on a dual coordinate frame to mitigate the coordinate inconsistencies among the stereo cameras and the IMUs. In the first stage, the IMU signals are transformed into the local frame and iteratively fused with the stereo images for estimating 3D poses in the local frame. In the second stage, the local poses are transformed into the global frame with the 6DOF head poses provided by the head-mounted display's (HMD) SLAM algorithm and then temporally aggregated via a temporal Transformer network. Meanwhile, to achieve geometric and semantic alignment among multi-modal features, we present a depth-guided spatial-aware deformable stereo attention network and a modality-aware Transformer decoder for cross-view and cross-modal feature fusion. Extensive experiments demonstrate that our approach achieves state-of-the-art performance on the public EMHI multi-modal egocentric pose estimation benchmark.

AAAI Conference 2026 Conference Paper

Scene Experts: Specializing in 3D Gaussian Splatting with Adaptive Decomposition

• Xiaowen Fu
• Yang Zhang
• Yuhan Tang
• Huazhong Zhang
• Tianxing Zhao
• Yuhang Guo
• Yu Huang
• Jinbao Wang

Anchor-based 3D Gaussian Splatting (GS), exemplified by Scaffold-GS, achieves remarkable storage efficiency through a hybrid explicit-implicit representation. However, their reliance on a single, monolithic network to decode anchor features imposes a severe bottleneck on model capacity, often resulting in blurred details and view-dependent artifacts in complex scenes. To break this bottleneck, we introduce the concept of Scene Experts: a strategy that decomposes the task of modeling a complex scene across a collection of specialized sub-models. To realize the paradigm, we propose MoE-GS. Our approach designs the decoder as a Sparsely-Gated Mixture of Experts (MoE), which dramatically increases the model's total capacity while maintaining comparable inference cost via sparse activation. To effectively train this high-capacity model, we propose two key innovations: (1) A progressive curriculum learning strategy that first trains all experts on a robust baseline before encouraging them to specialize on different scene components. (2) A novel opacity-aware regularization that penalizes inactive neural Gaussians, ensuring the expanded capacity is efficiently used. Extensive experiments demonstrate that MoE-GS substantially outperforms state-of-the-art methods on diverse benchmarks, significantly improving reconstruction fidelity while requiring a smaller or comparable Gaussian model size.

AAAI Conference 2026 Conference Paper

SL-CBM: Enhancing Concept Bottleneck Models with Semantic Locality for Better Interpretability

• Hanwei Zhang
• Luo Cheng
• Rui Wen
• Yang Zhang
• Lijun Zhang
• Holger Hermanns

Explainable AI (XAI) is crucial for building transparent and trustworthy machine learning systems, especially in high-stakes domains. Concept Bottleneck Models (CBMs) have emerged as a promising ante-hoc approach that provides interpretable, concept-level explanations by explicitly modeling human-understandable concepts. However, existing CBMs often suffer from poor locality faithfulness, failing to spatially align concepts with meaningful image regions, which limits their interpretability and reliability. In this work, we propose SL-CBM (CBM with Semantic Locality), a novel extension that enforces locality faithfulness by generating spatially coherent saliency maps at both concept and class levels. SL-CBM integrates a 1 × 1 convolutional layer with a cross-attention mechanism to enhance alignment between concepts, image regions, and final predictions. Unlike prior methods, SL-CBM produces faithful saliency maps inherently tied to the model’s internal reasoning, facilitating more effective debugging and intervention. Extensive experiments on image datasets demonstrate that SL-CBM substantially improves locality faithfulness, explanation quality, and intervention efficacy while maintaining competitive classification accuracy. Our ablation studies highlight the importance of contrastive and entropy-based regularization for balancing accuracy, sparsity, and faithfulness. Overall, SL-CBM bridges the gap between concept-based reasoning and spatial explainability, setting a new standard for interpretable and trustworthy concept-based models.

AAAI Conference 2026 Conference Paper

Subspace-Aware Graph Construction and Contrastive Alignment for Multimodal Recommendation with Large Language Models

• Haodong Li
• Lianyong Qi
• Weiming Liu
• Fan Wang
• Chong Li
• Shengye Pang
• Wenwen Gong
• Yanwei Xu

Multimedia content offers additional context for recommender systems to better understand user interests. Existing studies on multimodal recommendation primarily focus on constructing item-item semantic graphs. However, most of these methods capture only shallow semantic structures based on feature similarity and struggle to model more complex or cross-entity semantic relationships (e.g., user-item). Moreover, in these methods, collaborative signals often dominate and suppress semantic knowledge, which limits its role in representation learning. To address these issues, we propose SCALE, a novel framework that combines subspace-aware graph construction and contrastive alignment for multimodal recommendation with large language models. Specifically, we first use large language models and encoders to extract user and item features. Following the subspace clustering assumption, we apply the Orthogonal Matching Pursuit algorithm to mine complex semantic structures within the item-item, user-user, and user-item spaces, and integrate them into a unified semantic graph. We then perform graph convolution on both the semantic and interaction graphs, and aggregate the results for recommendation. Furthermore, contrastive losses are employed to enhance semantic fusion and alignment. Extensive experiments on five real-world datasets demonstrate that SCALE significantly outperforms state-of-the-art multimodal recommendation models, highlighting its effectiveness in modeling complex relationships and integrating semantic knowledge with collaborative signals.

TMLR Journal 2026 Journal Article

ThinkPrune: Pruning Long Chain-of-Thought of LLMs via Reinforcement Learning

• Bairu Hou
• Yang Zhang
• Jiabao Ji
• Yujian Liu
• Kaizhi Qian
• Jacob Andreas
• Shiyu Chang

We present ThinkPrune, a simple yet effective method for pruning the thinking length for long-thinking LLMs, which has been found to often produce inefficient and redundant thinking processes. Existing preliminary explorations of reducing thinking length primarily focus on forcing the thinking process to early exit, rather than adapting the LLM to optimize and consolidate the thinking process, and therefore the length-performance tradeoff observed so far is sub-optimal. To fill this gap, ThinkPrune offers a simple solution that continuously trains the long-thinking LLMs via reinforcement learning (RL) with an added token limit, beyond which any unfinished thoughts and answers will be discarded, resulting in a zero reward. To further preserve model performance, we introduce an iterative length pruning approach, where multiple rounds of RL are conducted, each with an increasingly more stringent token limit. We observed that ThinkPrune results in a remarkable performance-length tradeoff on the AIME24 dataset, the reasoning length of DeepSeek-R1-Distill-Qwen-1.5B can be reduced by half with only 2% drop in performance. We also observed that after pruning, the LLMs can bypass unnecessary steps while keeping the core reasoning process complete.

AAAI Conference 2026 Conference Paper

UDA: Unsupervised Debiasing Alignment for Pair-wise LLM-as-a-Judge

• Yang Zhang
• Cunxiang Wang
• Lindong Wu
• Wenbo Yu
• Yidong Wang
• Guangsheng Bao
• Jie Tang

Pairwise evaluation of Large Language Models (LLMs) is a common paradigm, but it is prone to preference bias, where judges systematically favor certain outputs, such as their own. This bias leads to inconsistent and skewed rankings across different judges. To address this, we first empirically demonstrate significant and heterogeneous biases in cross-model evaluations. We then propose UDA (Unsupervised Debiasing Alignment), a framework that reduces inter-judge disagreement by dynamically adjusting the Elo rating system. For each pairwise comparison, a compact neural network learns to adaptively set the K-factor and refine win probabilities. Crucially, UDA operates in a fully unsupervised manner, guided solely by the objective of minimizing the dispersion among the Elo trajectories of all judges. This forces an alignment towards a collective consensus, which serves as an unsupervised proxy for a more stable and reproducible evaluation. In addition, we provide theoretical motivation demonstrating how alignment towards a consensus can reduce aggregate system bias. Experiments show that UDA significantly reduces the inter-judge rating standard deviation by up to 63.4% and improves the average correlation with human judgments by 24.7%. Notably, UDA elevates the performance of poorly performing judges to achieve parity with high-quality ones, fostering a more robust and reliable evaluation ecosystem.

IS Journal 2025 Journal Article

A Generative Random Modality Dropout Framework for Robust Multimodal Emotion Recognition

• Yang Zhang
• Hui Chen
• Imad Rida
• Xianxun Zhu

Multimodal sentiment analysis faces significant challenges in real-world applications due to the frequent absence of modalities caused by privacy concerns, device limitations, or security policies. This article introduces a random modality dropout based on generative approach (RMDG), designed to enhance the robustness and performance of multimodal models under various modality absence scenarios. The RMDG method employs a generative approach during the training phase, where random modality dropout is applied to simulate missing modalities. By leveraging the remaining modalities to predict and regenerate the key features of the missing ones, the model effectively adapts to dynamic and unpredictable modality absences. This strategy not only eliminates the need for separate training or adjustments for each modality combination but also significantly improves the efficiency and accuracy of sentiment analysis in incomplete multimodal data scenarios. Extensive experiments demonstrate that RMDG outperforms existing methods, achieving superior performance in both complete and missing modality conditions.

IJCAI Conference 2025 Conference Paper

A Multimodal AI Dialogue System for Unified Document, Visual, and Audio Interaction

• Yujun Feng
• Jingyi Huang
• Yang Zhang

This paper presents a multimodal intelligent dialogue system that seamlessly integrates document analysis, visual media processing, and audio interaction within a unified web interface. The system ensures secure user identity verification through persistent conversational management, leveraging textual document analysis, dynamic context integration, and cross-media interactions via video, image, and real-time speech processing. Our approach introduces three key innovations: (1) context-aware document analysis through text extraction, (2) a multimodal input pipeline supporting images, videos, and audio, and (3) persistent chat history management for maintaining conversational continuity. The system facilitates seamless transitions between audio and text, enabling natural interactions by processing audio input and converting text responses into speech. Additionally, the platform provides an intuitive interface for document uploads, camera capture, and audio recording, while ensuring conversation context is preserved across sessions. This implementation demonstrates the practical integration of multimodal input in an interactive artificial intelligence (AI) system, showcasing its potential for enhanced user engagement and interaction.

NeurIPS Conference 2025 Conference Paper

Adjacent Words, Divergent Intents: Jailbreaking Large Language Models via Task Concurrency

• Yukun Jiang
• Mingjie Li
• Michael Backes
• Yang Zhang

Despite their superior performance on a wide range of domains, large language models (LLMs) remain vulnerable to misuse for generating harmful content, a risk that has been further amplified by various jailbreak attacks. Existing jailbreak attacks mainly follow sequential logic, where LLMs understand and answer each given task one by one. However, concurrency, a natural extension of the sequential scenario, has been largely overlooked. In this work, we first propose a word-level method to enable task concurrency in LLMs, where adjacent words encode divergent intents. Although LLMs maintain strong utility in answering concurrent tasks, which is demonstrated by our evaluations on mathematical and general question-answering benchmarks, we notably observe that combining a harmful task with a benign one significantly reduces the probability of it being filtered by the guardrail, showing the potential risks associated with concurrency in LLMs. Based on these findings, we introduce $\texttt{JAIL-CON}$, an iterative attack framework that $\underline{\text{JAIL}}$breaks LLMs via task $\underline{\text{CON}}$currency. Experiments on widely-used LLMs demonstrate the strong jailbreak capabilities of $\texttt{JAIL-CON}$ compared to existing attacks. Furthermore, when the guardrail is applied as a defense, compared to the sequential answers generated by previous attacks, the concurrent answers in our $\texttt{JAIL-CON}$ exhibit greater stealthiness and are less detectable by the guardrail, highlighting the unique feature of task concurrency in jailbreaking LLMs.

ICML Conference 2025 Conference Paper

AI for Global Climate Cooperation: Modeling Global Climate Negotiations, Agreements, and Long-Term Cooperation in RICE-N

• Tianyu Zhang
• Andrew Robert Williams
• Phillip Wozny
• Kai-Hendrik Cohrs
• Koen Ponse
• Marco Jiralerspong
• Soham R. Phade
• Sunil Srinivasa

Global cooperation on climate change mitigation is essential to limit temperature increases while supporting long-term, equitable economic growth and sustainable development. Achieving such cooperation among diverse regions, each with different incentives, in a dynamic environment shaped by complex geopolitical and economic factors, without a central authority, is a profoundly challenging game-theoretic problem. This article introduces RICE-N, a multi-region integrated assessment model that simulates the global climate, economy, and climate negotiations and agreements. RICE-N uses multi-agent reinforcement learning (MARL) to encourage agents to develop strategic behaviors based on the environmental dynamics and the actions of the others. We present two negotiation protocols: (1) Bilateral Negotiation, an exemplary protocol and (2) Basic Club, inspired from Climate Clubs and the carbon border adjustment mechanism (Nordhaus, 2015; Comissions, 2022). We compare their impact against a no-negotiation baseline with various mitigation strategies, showing that both protocols significantly reduce temperature growth at the cost of a minor drop in production while ensuring a more equitable distribution of the emission reduction costs.

EAAI Journal 2025 Journal Article

Attention-guided low-rank convolutional weighting for industrial missing data attacks

• Ming-Qing Zhang
• Liang-Yu Zhou
• Guo-Yu Liu
• Yan-Lin He
• Qun-Xiong Zhu
• Xin-Yi Cao
• Yang Zhang
• Yuan Xu

The widespread deployment of data acquisition units in industrial settings has significantly improved the efficiency of control and anomaly detection. However, when facing high proportions of missing data attacks, data quality often deteriorates, limiting the potential of data-driven optimization. To address this issue, we propose an attention-guided low-rank convolutional weighting method aimed at effectively estimating missing elements impacted by attacks. The proposed method combines attention mechanisms with convolutional unfolding techniques to fully exploit the low-rank characteristics of the data and faithfully reconstruct the missing data structure. Specifically, a low-rank convolutional weighting regularizer is introduced to capture low-rank subspace features of the original data and effectively guide the data recovery process. Meanwhile, an attention mechanism is used to accurately handle sparse noise, enhancing global robustness to sparse missing data. By employing iterative optimization, closed-form updates are achieved for solving variables and regularizers. Experimental results demonstrate that the proposed method provides advantages across several industrial application scenarios, including monoclonal antibody production in perfusion bioreactors using Chinese hamster ovary cells, multi-mode wastewater treatment plants, and image datasets. Even under high missing rates and non-uniform missing attack patterns, our method outperforms existing state-of-the-art methods.

AAAI Conference 2025 Conference Paper

Behavior Importance-Aware Graph Neural Architecture Search for Cross-Domain Recommendation

• Chendi Ge
• Xin Wang
• Ziwei Zhang
• Yijian Qin
• Hong Chen
• Haiyang Wu
• Yang Zhang
• Yuekui Yang

Cross-domain recommendation (CDR) mitigates data sparsity and cold-start issues in recommendation systems. While recent CDR approaches using graph neural networks (GNNs) capture complex user-item interactions, they rely on manually designed architectures that are often suboptimal and labor-intensive. Additionally, extracting valuable behavioral information from source domains to improve target domain recommendations remains challenging. To address these challenges, we propose Behavior importance-aware Graph Neural Architecture Search (BiGNAS), a framework that jointly optimizes GNN architecture and data importance for CDR. BiGNAS introduces two key components: a Cross-Domain Customized Supernetwork and a Graph-Based Behavior Importance Perceptron. The supernetwork, as a one-shot, retrain-free module, automatically searches the optimal GNN architecture for each domain without the need for retraining. The perceptron uses auxiliary learning to dynamically assess the importance of source domain behaviors, thereby improving target domain recommendations. Extensive experiments on benchmark CDR datasets and a large-scale industry advertising dataset demonstrate that BiGNAS consistently outperforms state-of-the-art baselines. To the best of our knowledge, this is the first work to jointly optimize GNN architecture and behavior data importance for cross-domain recommendation.

IJCAI Conference 2025 Conference Paper

Bidirectional Human–AI Collaboration for Equitable Student Performance Prediction via Deep Uncertainty Learning

• Ruohan Zong
• Yang Zhang
• Lanyu Shang
• Frank Stinar
• Nigel Bosch
• Dong Wang

This paper studies a bidirectional human-AI collaborative student performance prediction problem to enhance equitable online education, aligning with the United Nations' Sustainable Development Goal (SDG) of ensuring inclusive and equitable quality education for all. The goal is to leverage collaborative intelligence to generate accurate and fair student outcome predictions from behavioral data, ensuring equitable estimation for underrepresented populations. Current fair AI solutions often fail to mitigate demographic bias in the absence of student demographic data, while human-AI collaborative approaches frequently overlook human cognitive biases, leading to inaccurate predictions. We develop CollabDebias, a novel bidirectional human-AI collaborative framework that utilizes the complementary strengths of AI and humans to mitigate the AI demographic bias and human cognitive bias. To address AI demographic bias, we propose an uncertainty learning-based bias identification method and a reliability-aware human-AI integration approach. To reduce human cognitive bias, we design uncertainty-aware visualization of AI decision area and attention mechanism. Experimental results on an online course demonstrate CollabDebias's effectiveness in improving student performance prediction accuracy and fairness.

NeurIPS Conference 2025 Conference Paper

Can Large Multimodal Models Understand Agricultural Scenes? Benchmarking with AgroMind

• Qingmei Li
• Yang Zhang
• Zurong Mai
• Yuhang Chen
• Henglian Huang
• Jiarui Zhang
• Zhiwei Zhang
• Yibin Wen

Large Multimodal Models (LMMs) has demonstrated capabilities across various domains, but comprehensive benchmarks for agricultural remote sensing (RS) remain scarce. Existing benchmarks designed for agricultural RS scenarios exhibit notable limitations, primarily in terms of insufficient scene diversity in the dataset and oversimplified task design. To bridge this gap, we introduce AgroMind, a comprehensive agricultural remote sensing benchmark covering four task dimensions: spatial perception, object understanding, scene understanding, and scene reasoning, with a total of 13 task types, ranging from crop identification and health monitoring to environmental analysis. We curate a high-quality evaluation set by integrating nine public datasets and one private global parcel dataset, containing 28, 482 QA pairs and 20, 850 images. The pipeline begins with multi-source data pre-processing, including collection, format standardization, and annotation refinement. We then generate a diverse set of agriculturally relevant questions through the systematic definition of tasks. Finally, we employ LMMs for inference, generating responses, and performing detailed examinations. We evaluated 20 open-source LMMs and 4 closed-source models on AgroMind. Experiments reveal significant performance gaps, particularly in spatial reasoning and fine-grained recognition, it is notable that human performance lags behind several leading LMMs. By establishing a standardized evaluation framework for agricultural RS, AgroMind reveals the limitations of LMMs in domain knowledge and highlights critical challenges for future work. Data and code can be accessed at https: //rssysu. github. io/AgroMind/.

TMLR Journal 2025 Journal Article

Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models

• Yang Zhang
• Chenjia Bai
• Bin Zhao
• Junchi Yan
• Xiu Li
• Xuelong Li

Learning a world model for model-free Reinforcement Learning (RL) agents can significantly improve the sample efficiency by learning policies in imagination. However, building a world model for Multi-Agent RL (MARL) can be particularly challenging due to the scalability issue across different number of agents in a centralized architecture, and also the non-stationarity issue in a decentralized architecture stemming from the inter-dependency among agents. To address both challenges, we propose a novel world model for MARL that learns decentralized local dynamics for scalability, combined with a centralized representation aggregation from all agents. We cast the dynamics learning as an auto-regressive sequence modeling problem over discrete tokens by leveraging the expressive Transformer architecture, in order to model complex local dynamics across different agents and provide accurate and consistent long-term imaginations. As the first pioneering Transformer-based world model for multi-agent systems, we introduce a Perceiver Transformer as an effective solution to enable centralized representation aggregation within this context. Extensive results on Starcraft Multi-Agent Challenge (SMAC) and MAMujoco demonstrate superior sample efficiency and overall performance compared to strong model-free approaches and existing model-based methods.

NeurIPS Conference 2025 Conference Paper

DiffLiG: Diffusion-enhanced Liquid Graph with Attention Propagation for Grid-to-Station Precipitation Correction

• Yuxiang Li
• Yang Zhang
• Li Li
• Mengxuan Chen
• Meng Jin
• Fang Wang
• Haohuan Fu
• Juepeng Zheng

Modern precipitation forecasting systems, including reanalysis datasets, numerical models, and AI-based approaches, typically produce coarse-resolution gridded outputs. The process of converting these outputs to station-level predictions often introduces substantial spatial biases relative to station-level observations, especially in complex terrains or under extreme conditions. These biases stem from two core challenges: (i) $\textbf{station-level heterogeneity}$, with site-specific temporal and spatial dynamics; and (ii) $\textbf{oversmoothing}$, which blurs fine-scale variability in graph-based models. To address these issues, we propose $\textbf{DiffLiG}$ ($\underline{Diff}$usion-enhanced $\underline{Li}$quid $\underline{G}$raph with Attention Propagation), a graph neural network designed for precise spatial correction from gridded forecasts to station observations. DiffLiG integrates a GeoLiquidNet that adapts temporal encoding via site-aware OU dynamics, a graph neural network with a dynamic edge modulator that learns spatially adaptive connectivity, and a Probabilistic Diffusion Selector that generates and refines ensemble forecasts to mitigate oversmoothing. Experiments across multiple datasets show that DiffLiG consistently outperforms other methods, delivering more accurate and robust corrections across diverse geographic and climatic settings. Moreover, it achieves notable gains on other key meteorological variables, underscoring its generalizability and practical utility.

AAAI Conference 2025 Conference Paper

DoGA: Enhancing Grounded Object Detection via Grouped Pre-Training with Attributes

• Yang Liu
• Feng Hou
• Yunjie Peng
• Gangjian Zhang
• Yao Zhang
• Dong Xie
• Peng Wang
• Yang Zhang

Recent advances in vision-language pre-training have significantly enhanced the model capabilities on grounded object detection. However, these studies often pre-train with coarse-grained text prompts, such as plain category names and brief grounded phrases. This limitation curtails the model's capacity for fine-grained linguistic comprehension and leads to a significant decline in performance when faced with detailed descriptions or contextual information. To tackle these problems, we develop DoGA: Detect objects with Grouped Attributes, which employs commonly apparent attributes to bridge different granular semantics and uses specific attributes to identify the object discrepancy. Our DoGA incorporates three principle components: 1) Generation of attribute-based prompts, consisting of linguistic definitions enriched with common-sense visible attributes and hard negative notations deriving from the image-specific attribute features; 2) Paralleled entity fusion and optimization, designed to manage long attribute-based descriptions and negative concepts efficiently; and 3) Prompt-wise grouped training to accommodate model to perform many-to-many assignments, facilitating simultaneous training and inferring with multiple attribute-based synonyms. Extensive experiments demonstrate that training with synonymous attribute-based prompts allows DoGA to generalize multi-granular prompts and surpass previous state-of-the-art approaches, yielding 50.2 on the COCO and 38.0 on the LVIS benchmarks under the zero-short setting. We will make our code publicly available upon acceptance.

AAAI Conference 2025 Conference Paper

EMHI: A Multimodal Egocentric Human Motion Dataset with HMD and Body-Worn IMUs

• Zhen Fan
• Peng Dai
• Zhuo Su
• Xu Gao
• Zheng Lv
• Jiarui Zhang
• Tianyuan Du
• Guidong Wang

Egocentric human pose estimation (HPE) using wearable sensors is essential for VR/AR applications. Most methods rely solely on either egocentric-view images or sparse Inertial Measurement Unit (IMU) signals, leading to inaccuracies due to self-occlusion in images or the sparseness and drift of inertial sensors. Most importantly, the lack of real-world datasets containing both modalities is a major obstacle to progress in this field. To overcome the barrier, we propose EMHI, a multimodal Egocentric human Motion dataset with Head-Mounted Display (HMD) and body-worn IMUs, with all data collected under the real VR product suite. Specifically, EMHI provides synchronized stereo images from downward-sloping cameras on the headset and IMU data from body-worn sensors, along with pose annotations in SMPL format. This dataset consists of 885 sequences captured by 58 subjects performing 39 actions, totaling about 28.5 hours of recording. We evaluate the annotations by comparing them with optical marker-based SMPL fitting results. To substantiate the reliability of our dataset, we introduce MEPoser, a new baseline method for multimodal egocentric HPE, which employs a multimodal fusion encoder, temporal feature encoder, and MLP-based regression heads. The experiments on EMHI show that MEPoser outperforms existing single-modal methods and demonstrates the value of our dataset in solving the problem of egocentric HPE. We believe the release of EMHI and the method could advance the research of egocentric HPE and expedite the practical implementation of this technology in VR/AR products.

EAAI Journal 2025 Journal Article

Enhanced multi-modal emotion recognition using the feature level fusion

• Aziguli Wulamu
• Yuheng Wu
• Xin Liu
• Yao Zhang
• Jinghan Xu
• Yang Zhang

Multi-modal human emotion recognition is a complex process of synthesizing information from various modalities to calculate emotion states. This field faces several challenges: (1) Acoustic is an essential component of emotion expression, but it often underperforms compared to visual and text in emotion recognition. (2) Capturing the feature interaction among different modalities is usually complex. (3) Processing high-definition videos can significantly reduce the efficiency of visual analysis. In this study, we presented a learning architecture designed to recognize human emotions effectively. For the first challenge, we implemented a multi-level acoustic encoder (MLAE) that enhances the extraction of acoustic information to improve the acoustic contribution in multi-modal emotion recognition. Facing the second challenge, we introduced the cross-attention block module, which adeptly captures the inter-modal interactions. To address the third challenge, we adopted the re-parameterized visual geometry group network (RepVGG) as the visual feature encoder, employing its multi-branch learning and single-branch reasoning structure to maintain high reasoning efficiency. Our model has demonstrated the state-of-the-art performance of the interactive emotional dyadic motion capture (IEMOCAP) dataset and the multi-modal opinion sentiment and emotion intensity of the Carnegie Mellon University (CMU-MOSEI) dataset.

NeurIPS Conference 2025 Conference Paper

EnzyControl: Adding Functional and Substrate-Specific Control for Enzyme Backbone Generation

• Chao Song
• Zhiyuan Liu
• Han Huang
• Liang Wang
• Qiong Wang
• Jian-Yu Shi
• Hui Yu
• Yihang Zhou

Designing enzyme backbones with substrate-specific functionality is a critical challenge in computational protein engineering. Current generative models excel in protein design but face limitations in binding data, substrate-specific control, and flexibility for de novo enzyme backbone generation. To address this, we introduce EnzyBind, a dataset with 11, 100 experimentally validated enzyme-substrate pairs specifically curated from PDBbind. Building on this, we propose EnzyControl, a method that enables functional and substrate-specific control in enzyme backbone generation. Our approach generates enzyme backbones conditioned on MSA-annotated catalytic sites and their corresponding substrates, which are automatically extracted from curated enzyme-substrate data. At the core of EnzyControl is EnzyAdapter, a lightweight, modular component integrated into a pretrained motif-scaffolding model, allowing it to become substrate-aware. A two-stage training paradigm further refines the model's ability to generate accurate and functional enzyme structures. Experiments show that our EnzyControl achieves the best performance across structural and functional metrics on EnzyBind and EnzyBench benchmarks, with particularly notable improvements of 13% in designability and 13% in catalytic efficiency compared to the baseline models. The code is released at https: //github. com/Vecteur-libre/EnzyControl.

NeurIPS Conference 2025 Conference Paper

Finding and Reactivating Post-Trained LLMs' Hidden Safety Mechanisms

• Mingjie Li
• Wai Man Si
• Michael Backes
• Yang Zhang
• Yisen Wang

Despite the impressive performance of general-purpose large language models (LLMs), they often require fine-tuning or post-training to excel at specific tasks. For instance, large reasoning models (LRMs), such as the DeepSeek-R1 series, demonstrate strong reasoning capabilities after post-training different general large language models on diverse chain-of-thought (CoT) datasets. However, this additional training frequently comes at the cost of reduced safety, as the fine-tuned or post-trained models tend to exhibit more harmful behaviors compared with the regular LLMs before post-training or fine-tuning, potentially leading to harmful outcomes due to their enhanced capabilities. Taking LRMs as an example, we first investigate the underlying cause of this safety degradation in this paper. Our analysis reveals that post-training can mask the original safety mechanisms of the base LLM, while over-amplifying representations related to their post-training ability. But luckily, we also find that LRMs' safety mechanisms still exist instead of being removed during their post-training. Based on these findings, we propose a lightweight and cost-effective solution called SafeReAct that restores the suppressed safety behaviors by aligning with LoRA adapters on a few layers. Experiments on four state-of-the-art LRMs show that our method significantly improves safety on harmful prompts without compromising reasoning performance. Besides LRMs, additional results on other domain-specific LLMs, like medical models, further confirm the generality and effectiveness of our approach.

NeurIPS Conference 2025 Conference Paper

IGD: Token Decisiveness Modeling via Information Gain in LLMs for Personalized Recommendation

• Zijie Lin
• Yang Zhang
• Xiaoyan Zhao
• Fengbin Zhu
• Fuli Feng
• Tat-Seng Chua

Large Language Models (LLMs) have shown strong potential for recommendation by framing item prediction as a token-by-token language generation task. However, existing methods treat all item tokens equally, simply pursuing likelihood maximization during both optimization and decoding. This overlooks crucial token-level differences in decisiveness—many tokens contribute little to item discrimination yet can dominate optimization or decoding. To quantify token decisiveness, we propose a novel perspective that models item generation as a decision process, measuring token decisiveness by the Information Gain (IG) each token provides in reducing uncertainty about the generated item. Our empirical analysis reveals that most tokens have low IG but often correspond to high logits, disproportionately influencing training loss and decoding, which may impair model performance. Building on these insights, we introduce an Information Gain-based Decisiveness-aware Token handling (IGD) strategy that integrates token decisiveness into both tuning and decoding. Specifically, IGD downweights low-IG tokens during tuning and rebalances decoding to emphasize tokens with high IG. In this way, IGD moves beyond pure likelihood maximization, effectively prioritizing high-decisiveness tokens. Extensive experiments on four benchmark datasets with two LLM backbones demonstrate that IGD consistently improves recommendation accuracy, achieving significant gains on widely used ranking metrics compared to strong baselines. Our codes are available at \url{https: //github. com/ZJLin2oo1/IGD}.

EAAI Journal 2025 Journal Article

Latent temporal smoothness-induced Schatten- p norm factorization for sequential subspace clustering

• Yuan Xu
• Zhen-Zhen Zhao
• Tong-Wei Lu
• Wei Ke
• Yi Luo
• Yan-Lin He
• Qun-Xiong Zhu
• Yang Zhang

This paper presents an innovative latent temporal smoothness-induced Schatten- p norm factorization (SpFLTS) method aimed at addressing challenges in sequential subspace clustering tasks. Globally, SpFLTS employs a low-rank subspace clustering framework based on Schatten-2/3 norm factorization to enhance the comprehensive capture of the original data features. Locally, a total variation smoothing term is induced to the temporal gradients of latent subspace matrices obtained from sub-orthogonal projections, thereby preserving smoothness in the sequential latent space. To efficiently solve the closed-form optimization problem, a fast Fourier transform is combined with the non-convex alternating direction method of multipliers to optimize latent subspace matrix, which greatly speeds up computation. Experimental results demonstrate that the proposed SpFLTS method surpasses existing techniques on multiple benchmark databases, highlighting its superior clustering performance and extensive application potential.

NeurIPS Conference 2025 Conference Paper

Learning 3D Anisotropic Noise Distributions Improves Molecular Force Fields

• Xixian Liu
• Rui Jiao
• Zhiyuan Liu
• Yurou Liu
• Yang Liu
• Ziheng Lu
• Wenbing Huang
• Yang Zhang

Coordinate denoising has emerged as a promising method for 3D molecular pretraining due to its theoretical connection to learning molecular force field. However, existing denoising methods rely on oversimplied molecular dynamics that assume atomic motions to be isotropic and homoscedastic. To address these limitations, we propose a novel denoising framework AniDS: Anisotropic Variational Autoencoder for 3D Molecular Denoising. AniDS introduces a structure-aware anisotropic noise generator that can produce atom-specific, full covariance matrices for Gaussian noise distributions to better reflect directional and structural variability in molecular systems. These covariances are derived from pairwise atomic interactions as anisotropic corrections to an isotropic base. Our design ensures that the resulting covariance matrices are symmetric, positive semi-definite, and SO(3)-equivariant, while providing greater capacity to model complex molecular dynamics. Extensive experiments show that AniDS outperforms prior isotropic and homoscedastic denoising models and other leading methods on the MD17 and OC22 benchmarks, achieving average relative improvements of 8. 9% and 6. 2% in force prediction accuracy. Our case study on a crystal and molecule structure shows that AniDS adaptively suppresses noise along the bonding direction, consistent with physicochemical principles. Our code is available at https: //github. com/ZeroKnighting/AniDS.

IJCAI Conference 2025 Conference Paper

Leveraging MLLM Embeddings and Attribute Smoothing for Compositional Zero-Shot Learning

• Xudong Yan
• Songhe Feng
• Yang Zhang
• Jian Yang
• Yueguan Lin
• Haojun Fei

Compositional zero-shot learning (CZSL) aims to recognize novel compositions of attributes and objects learned from seen compositions. Previous works disentangle attributes and objects by extracting shared and exclusive parts between the image pair sharing the same attribute (object), as well as aligning them with pretrained word embeddings to improve unseen attribute-object recognition. Despite the significant achievements of existing efforts, they are hampered by three limitations: (1) The efficacy of disentanglement is compromised due to the influence of the background and the intricate entanglement of attributes with objects in the same parts. (2) Existing word embeddings fail to capture complex multimodal semantic information. (3) Overconfidence exhibited by existing models in seen compositions hinders their generalization to novel compositions. Being aware of these, we propose a novel framework named multimodal large language model (MLLM) embeddings and attribute smoothing guided disentanglement for CZSL. First, we leverage feature adaptive aggregation modules to mitigate the impact of background, and utilize learnable condition masks to capture multi-granularity features for disentanglement. Moreover, the last hidden states of MLLM are employed as word embeddings for their superior representation capabilities. Furthermore, we propose attribute smoothing with auxiliary attributes generated by the large language model (LLM) for seen compositions to address the overconfidence challenge. Extensive experiments demonstrate that our method achieves state-of-the-art performance on three challenging datasets. The supplementary material and source code will be available at https: //github. com/xud-yan/Trident.

ICML Conference 2025 Conference Paper

Minimalist Concept Erasure in Generative Models

• Yang Zhang
• Er Jin
• Yanfei Dong
• Yixuan Wu
• Philip H. S. Torr
• Ashkan Khakzar
• Johannes Stegmaier
• Kenji Kawaguchi

Recent advances in generative models have demonstrated remarkable capabilities in producing high-quality images, but their reliance on large-scale unlabeled data has raised significant safety and copyright concerns. Efforts to address these issues by erasing unwanted concepts have shown promise. However, many existing erasure methods involve excessive modifications that compromise the overall utility of the model. In this work, we address these issues by formulating a novel minimalist concept erasure objective based only on the distributional distance of final generation outputs. Building on our formulation, we derive a tractable loss for differentiable optimization that leverages backpropagation through all generation steps in an end-to-end manner. We also conduct extensive analysis to show theoretical connections with other models and methods. To improve the robustness of the erasure, we incorporate neuron masking as an alternative to model fine-tuning. Empirical evaluations on state-of-the-art flow-matching models demonstrate that our method robustly erases concepts without degrading overall model performance, paving the way for safer and more responsible generative models.

IROS Conference 2025 Conference Paper

Minimizing Acoustic Noise: Enhancing Quiet Locomotion for Quadruped Robots in Indoor Applications

• Zhanxiang Cao
• Buqing Nie
• Yang Zhang
• Yue Gao 0005

Recent advancements in quadruped robot research have significantly improved their ability to traverse complex and unstructured outdoor environments. However, the issue of noise generated during locomotion is generally overlooked, which is critically important in noise-sensitive indoor environments, such as service and healthcare settings, where maintaining low noise levels is essential. This study aims to optimize the acoustic noise generated by quadruped robots during locomotion through the development of advanced motion control algorithms. To achieve this, we propose a novel approach that minimizes noise emissions by integrating optimized gait design with tailored control strategies. This method achieves an average noise reduction of approximately 8 dBA during movement, thereby enhancing the suitability of quadruped robots for deployment in noise-sensitive indoor environments. Experimental results demonstrate the effectiveness of this approach across various indoor settings, highlighting the potential of quadruped robots for quiet operation in noise-sensitive environments.

EAAI Journal 2025 Journal Article

Omni-scale spatio-temporal attention network for impact localization of sandwich composite panels

• Yang Zhang
• Bo Yang
• Shilong Wang
• Fan Mo
• Fengyang Bi
• Yan He

Sandwich composite panels (SCPs) have been widely used in aerospace, shipbuilding, and other fields due to their excellent flexural rigidity, designability, and other advantages. However, SCPs are susceptible to damage from external impacts during service, resulting in internal damage that is difficult to detect, and seriously affecting the equipment's structural performance and service life. Accurate impact localization is a prerequisite for condition monitoring and damage detection of SCPs. However, existing methods cannot achieve accurate impact localization of SCPs since the core material of SCPs absorbs impact energy and suppresses signal transmission. Hence, this paper investigates the utilization of deep learning technology to analyze impact signals, aiming to achieve accurate localization. Firstly, an impact signal acquisition scheme is designed. Then, this paper designs an omni-scale spatio-temporal attention network (OSTNet). It has the capability to extract all temporal scale features and spatio-temporal correlations of impact signals, thereby effectively handling SCPs impact signals with concentrated energy and low discriminability. Moreover, OSTNet also processes the impact signals in frequency domain to further improve accuracy. Finally, we set up the experimental platform and established an impact localization dataset of SCPs to test the localization effect of OSTNet. Experimental results show that the maximum, minimum, and average location errors of OSTNet are 7. 410 mm, 1. 145 mm, and 5. 157 mm, respectively, which are 11. 93 mm, 1. 776 mm, and 4. 372 mm lower than the optimal values of classical comparison models SFNet, PZTNet, LSTM, and RNN, respectively. And, OSTNet outperforms the recently published MNQGN and DAGNN methods, with 2. 378 mm, 5. 070 mm, and 1. 070 mm decreasing, respectively. Moreover, the statistical tests also strongly confirm that OSTNet exhibits significant differences compared to the 6 comparative networks. Furthermore, the effectiveness of OSTNet is validated by employing impact signal data with varying energy levels, resulting in satisfactory localization performances being achieved.

ICLR Conference 2025 Conference Paper

Online Preference Alignment for Language Models via Count-based Exploration

• Chenjia Bai
• Yang Zhang
• Shuang Qiu
• Qiaosheng Zhang 0002
• Kang Xu
• Xuelong Li 0001

Reinforcement Learning from Human Feedback (RLHF) has shown great potential in fine-tuning Large Language Models (LLMs) to align with human preferences. Existing methods perform preference alignment from a fixed dataset, which can be limited in data coverage and the resulting reward model is hard to generalize in out-of-distribution responses. Thus, online RLHF is more desirable to empower the LLM to explore outside the support of the initial dataset by iteratively collecting the prompt-response pairs. In this paper, we study the fundamental problem in online RLHF, i.e., how to explore for LLM. We give a theoretical motivation in linear reward assumption to show that an optimistic reward with an upper confidence bound (UCB) term leads to a provably efficient RLHF policy. Then, we reformulate our objective to direct preference optimization with an exploration term, where the UCB-term can be converted to a count-based exploration bonus. We further propose a practical algorithm, named Count-based Online Preference Optimization (COPO), which leverages a simple coin-flip counting module to estimate the pseudo-count of a prompt-response pair in previously collected data. COPO encourages LLMs to balance exploration and preference optimization in an iterative manner, which enlarges the exploration space and the entire data coverage of iterative LLM policies. We conduct online RLHF experiments on Zephyr and Llama-3 models. The results on instruction-following and standard academic benchmarks show that COPO significantly increases performance.

IROS Conference 2025 Conference Paper

PANDAS: Prediction and Detection of Accurate Slippage

• Teng Yan
• Xiaohong Zhou
• Jiamin Long
• Wenxian Li
• Yang Zhang

High-resolution tactile sensing and advanced computational models have accelerated progress in robotic grasping; however, real-time, stable manipulation of smooth and fragile objects still lags behind. The challenges are twofold: first, the robot must detect incipient slip at sub-millimeter scales in real time; second, the system must issue millisecond-level early warnings before true instability occurs so that the controller has sufficient time to react. To address these challenges, we propose PANDAS (Prediction AND Detection of Accurate Slippage), a framework that integrates a physics-informed, multimodal spatiotemporal network for slip detection with a probabilistic temporal reasoning module for forecasting near-future risk. Experimental results demonstrate that the proposed method achieves a slip sensitivity of 94. 6%, a response latency of 28ms, and an early-warning lead time of 32ms. Moreover, under 5dB Gaussian noise, it maintains a high F1-score of 92. 3%, validating its robustness, predictive capability, and suitability for edge deployment in dynamic, high-noise environments.

EAAI Journal 2025 Journal Article

Prediction of high-performance concrete compressive strength using Decision Tree-Guided Artificial Neural Network Pretraining approach

• Yang Zhang
• Xinghai Yuan
• Xuanpeng Zhang
• Heng Wang
• Pan He
• Ling Luo
• Chuanyun Xu

Deep learning models exhibit substantial nonlinear learning capabilities, enabling them to predict the compressive strength of high-performance concrete based on its mix proportions and to discern the intricate nonlinear relationships between these proportions and compressive strength. However, the data-hungry nature of deep learning necessitates extensive training datasets on mix proportions, and generating such large-scale, high-quality concrete mix datasets incurs significant costs. To address this issue, this study proposes a Decision Tree-guided Artificial Neural Network Pretraining (TANNP) approach, aimed at training high-accuracy concrete compressive strength prediction models using small-scale datasets. This approach synergizes the advantages of decision trees in managing small datasets and the powerful learning capabilities of neural networks. The TANNP method was experimentally validated on a concrete compressive strength dataset. The results demonstrate that compared to directly employing neural networks, the TANNP approach significantly improves performance metrics: the coefficient of determination increased from 0. 91 to 0. 97, the root mean square error decreased from 5. 10 MPa to 3. 26 MPa (a reduction of approximately 36. 1%), the mean absolute error was reduced from 3. 00 MPa to 2. 10 MPa, the mean absolute percentage error dropped from 10. 72% to 7. 24%, and the A20-Index improved from 0. 88 to 0. 92, achieving state-of-the-art results on this dataset. These findings indicate that the TANNP approach effectively mitigates the data-hunger issue inherent in deep learning models and significantly enhances the accuracy of concrete compressive strength predictions.

NeurIPS Conference 2025 Conference Paper

PRING: Rethinking Protein-Protein Interaction Prediction from Pairs to Graphs

• Xinzhe Zheng
• Hao Du
• Fanding Xu
• Jinzhe Li
• Zhiyuan Liu
• Wenkang Wang
• Tao Chen
• Wanli Ouyang

Deep learning-based computational methods have achieved promising results in predicting protein-protein interactions (PPIs). However, existing benchmarks predominantly focus on isolated pairwise evaluations, overlooking a model's capability to reconstruct biologically meaningful PPI networks, which is crucial for biology research. To address this gap, we introduce PRING, the first comprehensive benchmark that evaluates PRotein-protein INteraction prediction from a Graph-level perspective. PRING curates a high-quality, multi-species PPI network dataset comprising 21, 484 proteins and 186, 818 interactions, with well-designed strategies to address both data redundancy and leakage. Building on this golden-standard dataset, we establish two complementary evaluation paradigms: (1) topology-oriented tasks, which assess intra and cross-species PPI network construction, and (2) function-oriented tasks, including protein complex pathway prediction, GO module analysis, and essential protein justification. These evaluations not only reflect the model's capability to understand the network topology but also facilitate protein function annotation, biological module detection, and even disease mechanism analysis. Extensive experiments on four representative model categories, consisting of sequence similarity-based, naive sequence-based, protein language model-based, and structure-based approaches, demonstrate that current PPI models have potential limitations in recovering both structural and functional properties of PPI networks, highlighting the gap in supporting real-world biological applications. We believe PRING provides a reliable platform to guide the development of more effective PPI prediction models for the community. The dataset and source code of PRING are available at https: //github. com/SophieSarceau/PRING.

NeurIPS Conference 2025 Conference Paper

ProteinConformers: Benchmark Dataset for Simulating Protein Conformational Landscape Diversity and Plausibility

• Yihang Zhou
• Chen Wei
• Minghao Sun
• Jin Song
• Yang Li
• Lin Wang
• Yang Zhang

Understanding the conformational landscape of proteins is essential for elucidating protein function and facilitating drug design. However, existing protein conformation benchmarks fail to capture the full energy landscape, limiting their ability to evaluate the diversity and physical plausibility of AI-generated structures. We introduce ProteinConformers, a large-scale benchmark dataset comprising over 381, 000 physically realistic conformations for 87 CASP targets. These were derived from more than 40, 000 structural decoys via extensive all-atom molecular dynamics simulations totaling over 6 million CPU hours. Using this dataset, we propose novel metrics to evaluate conformational diversity and plausibility, and systematically benchmark six protein conformation generative models. Our results highlight that leveraging large-scale protein sequence data can enhance a model’s ability to explore conformational space, potentially reducing reliance on MD-derived data. Additionally, we find that PDB and MD datasets influence model performance differently, current models perform well on inter-atomic distance prediction but struggle with inter-residue orientation generation. Overall, our dataset, evaluation metrics, and benchmarking results provide the first comprehensive foundation for assessing generative models in protein conformational modeling. Dataset and instructions are available at https: //huggingface. co/ datasets/Jim990908/ProteinConformers/tree/main. Codes are stored at https: //github. com/auroua/ProteinConformers. An interactive website locates at https: //zhanggroup. org/ProteinConformers.

ICML Conference 2025 Conference Paper

Q-Supervised Contrastive Representation: A State Decoupling Framework for Safe Offline Reinforcement Learning

• Zhihe Yang
• Yunjian Xu
• Yang Zhang

Safe offline reinforcement learning (RL), which aims to learn the safety-guaranteed policy without risky online interaction with environments, has attracted growing recent attention for safety-critical scenarios. However, existing approaches encounter out-of-distribution problems during the testing phase, which can result in potentially unsafe outcomes. This issue arises due to the infinite possible combinations of reward-related and cost-related states. In this work, we propose State Decoupling with Q-supervised Contrastive representation (SDQC), a novel framework that decouples the global observations into reward- and cost-related representations for decision-making, thereby improving the generalization capability for unfamiliar global observations. Compared with the classical representation learning methods, which typically require model-based estimation (e. g. , bisimulation), we theoretically prove that our Q-supervised method generates a coarser representation while preserving the optimal policy, resulting in improved generalization performance. Experiments on DSRL benchmark problems provide compelling evidence that SDQC surpasses other baseline algorithms, especially for its exceptional ability to achieve almost zero violations in more than half of the tasks, while the state-of-the-art algorithm can only achieve the same level of success in a quarter of the tasks. Further, we demonstrate that SDQC possesses superior generalization ability when confronted with unseen environments.

NeurIPS Conference 2025 Conference Paper

QiMeng-SALV: Signal-Aware Learning for Verilog Code Generation

• Yang Zhang
• Rui Zhang
• Jiaming Guo
• Huang Lei
• Di Huang
• Yunpu Zhao
• Shuyao Cheng
• Pengwei Jin

The remarkable progress of Large Language Models (LLMs) presents promising opportunities for Verilog code generation which is significantly important for automated circuit design. The lacking of meaningful functional rewards hinders the preference optimization based on Reinforcement Learning (RL) for producing functionally correct Verilog code. In this paper, we propose Signal-Aware Learning for Verilog code generation (QiMeng-SALV) by leveraging code segments of functionally correct output signal to optimize RL training. Considering Verilog code specifies the structural interconnection of hardware gates and wires so that different output signals are independent, the key insight of QiMeng-SALV is to extract verified signal-aware implementations in partially incorrect modules, so as to enhance the extraction of meaningful functional rewards. Roughly, we verify the functional correctness of signals in generated module by comparing with that of reference module in the training data. Then abstract syntax tree (AST) is employed to identify signal-aware code segments which can provide meaningful functional rewards from erroneous modules. Finally, we introduce signal-aware DPO which is optimized on the correct signal-level code segments, thereby preventing noise and interference from incorrect signals. The proposed QiMeng-SALV underscores the paradigm shift from conventional module-level to fine-grained signal-level optimization in Verilog code generation, addressing the issue of insufficient functional rewards. Experiments demonstrate that our method achieves state-of-the-art performance on VerilogEval and RTLLM, with a 7B parameter model matching the performance of the DeepSeek v3 671B model and significantly outperforming the leading open-source model CodeV trained on the same dataset.

NeurIPS Conference 2025 Conference Paper

Revisiting Multi-Agent World Modeling from a Diffusion-Inspired Perspective

• Yang Zhang
• Xinran Li
• Jianing Ye
• Shuang Qiu
• Delin Qu
• Xiu Li
• Chongjie Zhang
• Chenjia Bai

World models have recently attracted growing interest in Multi-Agent Reinforcement Learning (MARL) due to their ability to improve sample efficiency for policy learning. However, accurately modeling environments in MARL is challenging due to the exponentially large joint action space and highly uncertain dynamics inherent in multi-agent systems. To address this, we reduce modeling complexity by shifting from jointly modeling the entire state-action transition dynamics to focusing on the state space alone at each timestep through sequential agent modeling. Specifically, our approach enables the model to progressively resolve uncertainty while capturing the structured dependencies among agents, providing a more accurate representation of how agents influence the state. Interestingly, this sequential revelation of agents' actions in a multi-agent system aligns with the reverse process in diffusion models—a class of powerful generative models known for their expressiveness and training stability compared to autoregressive or latent variable models. Leveraging this insight, we develop a flexible and robust world model for MARL using diffusion models. Our method, \textbf{D}iffusion-\textbf{I}nspired \textbf{M}ulti-\textbf{A}gent world model (DIMA), achieves state-of-the-art performance across multiple multi-agent control benchmarks, significantly outperforming prior world models in terms of final return and sample efficiency, including MAMuJoCo and Bi-DexHands. DIMA establishes a new paradigm for constructing multi-agent world models, advancing the frontier of MARL research. Codes are open-sourced at \url{https: //github. com/breez3young/DIMA}.

AAAI Conference 2025 Conference Paper

SIDE: Socially Informed Drought Estimation Toward Understanding Societal Impact Dynamics of Environmental Crisis

• Lanyu Shang
• Bozhang Chen
• Shiwei Liu
• Yang Zhang
• Ruohan Zong
• Anav Vora
• Ximing Cai
• Na Wei

Drought has become a critical global threat with significant societal impact. Existing drought monitoring solutions primarily focus on assessing drought severity using quantitative measurements, overlooking the diverse societal impact of drought from human-centric perspectives. Motivated by the collective intelligence on social media and the computational power of AI, this paper studies a novel problem of socially informed AI-driven drought estimation that aims to leverage social and news media information to jointly estimate drought severity and its societal impact. Two technical challenges exist: 1) How to model the implicit temporal dynamics of drought societal impact. 2) How to capture the social-physical interdependence between the physical drought condition and its societal impact. To address these challenges, we develop SIDE, a socially informed AI-driven drought estimation framework that explicitly quantifies the societal impact of drought and effectively models the social-physical interdependency for joint severity-impact estimation. Experiments on real-world datasets from California and Texas demonstrate SIDE's superior performance compared to state-of-the-art baselines in accurately estimating drought severity and its societal impact. SIDE offers valuable insights for developing human-centric drought mitigation strategies to foster sustainable and resilient communities.

ICML Conference 2025 Conference Paper

Task-Agnostic Pre-training and Task-Guided Fine-tuning for Versatile Diffusion Planner

• Chenyou Fan
• Chenjia Bai
• Zhao Shan
• Haoran He
• Yang Zhang
• Zhen Wang 0004

Diffusion models have demonstrated their capabilities in modeling trajectories of multi-tasks. However, existing multi-task planners or policies typically rely on task-specific demonstrations via multi-task imitation, or require task-specific reward labels to facilitate policy optimization via Reinforcement Learning (RL). They are costly due to the substantial human efforts required to collect expert data or design reward functions. To address these challenges, we aim to develop a versatile diffusion planner capable of leveraging large-scale inferior data that contains task-agnostic sub-optimal trajectories, with the ability to fast adapt to specific tasks. In this paper, we propose SODP, a two-stage framework that leverages Sub-Optimal data to learn a Diffusion Planner, which is generalizable for various downstream tasks. Specifically, in the pre-training stage, we train a foundation diffusion planner that extracts general planning capabilities by modeling the versatile distribution of multi-task trajectories, which can be sub-optimal and has wide data coverage. Then for downstream tasks, we adopt RL-based fine-tuning with task-specific rewards to quickly refine the diffusion planner, which aims to generate action sequences with higher task-specific returns. Experimental results from multi-task domains including Meta-World and Adroit demonstrate that SODP outperforms state-of-the-art methods with only a small amount of data for reward-guided fine-tuning.

NeurIPS Conference 2025 Conference Paper

The Emergence of Abstract Thought in Large Language Models Beyond Any Language

• Yuxin Chen
• Yiran Zhao
• Yang Zhang
• An Zhang
• Kenji Kawaguchi
• Shafiq Joty
• Junnan Li
• Tat-Seng Chua

As large language models (LLMs) continue to advance, their capacity to function effectively across a diverse range of languages has shown marked improvement. Preliminary studies observe that the hidden activations of LLMs often resemble English, even when responding to non-English prompts. This has led to the widespread assumption that LLMs may ``think'' in English. However, more recent results showing strong multilingual performance, even surpassing English performance on specific tasks in other languages, challenge this view. In this work, we find that LLMs progressively develop a core language-agnostic parameter space—a remarkably small subset of parameters whose deactivation results in significant performance degradation across all languages. This compact yet critical set of parameters underlies the model’s ability to generalize beyond individual languages, supporting the emergence of abstract thought that is not tied to any specific linguistic system. Specifically, we identify language-related neurons—those are consistently activated during the processing of particular languages, and categorize them as either shared (active across multiple languages) or exclusive (specific to one). As LLMs undergo continued development over time, we observe a marked increase in both the proportion and functional importance of shared neurons, while exclusive neurons progressively diminish in influence. These shared neurons constitute the backbone of the core language-agnostic parameter space, supporting the emergence of abstract thought. Motivated by these insights, we propose neuron-specific training strategies tailored to LLMs' language-agnostic levels at different development stages. Experiments across diverse LLM families support our approach. Our codes are available at https: //anonymous. 4open. science/status/S-C393.

IJCAI Conference 2025 Conference Paper

Token-Level Accept or Reject: A Micro Alignment Approach for Large Language Models

• Yang Zhang
• Yu Yu
• Bo Tang
• Yu Zhu
• Chuxiong Sun
• Wenqiang Wei
• Jie Hu
• Zipeng Xie

With the rapid development of Large Language Models (LLMs), aligning these models with human preferences and values is critical to ensuring ethical and safe applications. However, existing alignment techniques such as RLHF or DPO often require direct fine-tuning on LLMs with billions of parameters, resulting in substantial computational costs and inefficiencies. To address this, we propose Micro token-level Accept-Reject Aligning (MARA) approach designed to operate independently of the language models. MARA simplifies the alignment process by decomposing sentence-level preference learning into token-level binary classification, where a compact three-layer fully-connected network determines whether candidate tokens are “Accepted” or “Rejected” as part of the response. Extensive experiments across seven different LLMs and three open-source datasets show that MARA achieves significant improvements in alignment performance while reducing computational costs. The source code and implementation details are publicly available at https: //github. com/IAAR-Shanghai/MARA, and the trained models are released at https: //huggingface. co/IAAR-Shanghai/MARA_AGENTS.

NeurIPS Conference 2025 Conference Paper

Towards Unified and Lossless Latent Space for 3D Molecular Latent Diffusion Modeling

• Yanchen Luo
• Zhiyuan Liu
• Yi Zhao
• Sihang Li
• Hengxing Cai
• Kenji Kawaguchi
• Tat-Seng Chua
• Yang Zhang

3D molecule generation is crucial for drug discovery and material science, requiring models to process complex multi-modalities, including atom types, chemical bonds, and 3D coordinates. A key challenge is integrating these modalities of different shapes while maintaining SE(3) equivariance for 3D coordinates. To achieve this, existing approaches typically maintain separate latent spaces for invariant and equivariant modalities, reducing efficiency in both training and sampling. In this work, we propose U nified Variational A uto- E ncoder for 3D Molecular Latent Diffusion Modeling ( UAE-3D ), a multi-modal VAE that compresses 3D molecules into latent sequences from a unified latent space, while maintaining near-zero reconstruction error. This unified latent space eliminates the complexities of handling multi-modality and equivariance when performing latent diffusion modeling. We demonstrate this by employing the Diffusion Transformer--a general-purpose diffusion model without any molecular inductive bias--for latent generation. Extensive experiments on GEOM-Drugs and QM9 datasets demonstrate that our method significantly establishes new benchmarks in both de novo and conditional 3D molecule generation, achieving leading efficiency and quality. On GEOM-Drugs, it reduces FCD by 72. 6% over the previous best result, while achieving over 70% relative average improvements in geometric fidelity. Our code is released at https: //github. com/lyc0930/UAE-3D/.

AAAI Conference 2025 Conference Paper

VIoTGPT: Learning to Schedule Vision Tools Towards Intelligent Video Internet of Things

• Yaoyao Zhong
• Mengshi Qi
• Rui Wang
• Yuhan Qiu
• Yang Zhang
• Huadong Ma

Video Internet of Things (VIoT) has shown full potential in collecting an unprecedented volume of video data. How to schedule the domain-specific perceiving models and analyze the collected videos uniformly, efficiently, and especially intelligently to accomplish complicated tasks is challenging. To address the challenge, we build VIoTGPT, the framework based on LLMs to correctly interact with humans, query knowledge videos, and invoke vision models to analyze multimedia data collaboratively. To support VIoTGPT and related future works, we meticulously crafted the VIoT-Tool dataset, including the training dataset and the benchmark involving 11 representative vision models across three categories based on semi-automatic annotations. To guide LLM to act as the intelligent agent towards intelligent VIoT, we resort to ReAct instruction tuning method based on VIoT-Tool to learn the tool capability. Quantitative and qualitative experiments and analyses demonstrate the effectiveness of VIoTGPT. We believe VIoTGPT contributes to improving human-centered experiences in VIoT applications.

TMLR Journal 2024 Journal Article

A Dual-Perspective Approach to Evaluating Feature Attribution Methods

• Yawei Li
• Yang Zhang
• Kenji Kawaguchi
• Ashkan Khakzar
• Bernd Bischl
• Mina Rezaei

Feature attribution methods attempt to explain neural network predictions by identifying relevant features. However, establishing a cohesive framework for assessing feature attribution remains a challenge. There are several views through which we can evaluate attributions. One principal lens is to observe the effect of perturbing attributed features on the model’s behavior (i.e., faithfulness). While providing useful insights, existing faithfulness evaluations suffer from shortcomings that we reveal in this paper. To address the limitations of previous evaluations, in this work, we propose two new perspectives within the faithfulness paradigm that reveal intuitive properties: soundness and completeness. Soundness assesses the degree to which attributed features are truly predictive features, while completeness examines how well the resulting attribution reveals all the predictive features. The two perspectives are based on a firm mathematical foundation and provide quantitative metrics that are computable through efficient algorithms. We apply these metrics to mainstream attribution methods, offering a novel lens through which to analyze and compare feature attribution methods.

YNIMG Journal 2024 Journal Article

A first-in-human application of OPM-MEG for localizing motor activity area: Compared to functional MRI

• Tai Sun
• Xiaohan Chi
• Yuming Peng
• Qianhe Zhang
• Kang Liu
• Yiwen Ma
• Ming Ding
• Nan Ji

BACKGROUND: Accurately localizing brain motor areas is vital for protecting motor function during neurosurgical procedures. Magnetoencephalography (MEG) based on optically pumped magnetometer (OPM) improves the availability of MEG in clinical applications. The aim of this study is to evaluate the availability, accuracy and precision of "OPM-MEG" for localizing motor areas in brain tumor patients and healthy individuals. METHODS: Participants were enrolled and subjected to primary motor area localization by both 3T-fMRI and 128-channel OPM-MEG examinations. The localization accuracy (ability of mapping on the anatomical location) and precision (activation signal centralization) were compared between the two methods, and accuracy was further validated by intraoperative direct cortical electrical stimulation (DCS) on the localized area with assistance of neuro-navigation system. RESULT: A total of 12 participants (7 brain tumor patients and 5 healthy individuals) were enrolled and all had successful localization for motor areas by both methods. The average time of OPM-MEG examination for each limb function was approximately 9 min. The localizations by both methods mainly covered the anatomical location of primary motor cortex and were partially overlapped. The motor activation signal identified by OPM-MEG was more centralized than fMRI did. The centroid of motor area localized by the OPM-MEG deviated from it by fMRI, with a mean distance of 19.7 mm and 27.48 mm for hand or foot localization, respectively. Furthermore, the OPM-MEG centroid for hand movement successfully triggered corresponding hand response by DCS. CONCLUSIONS: In this first-in-human study exploring the potential of OPM-MEG in functional localization of motor areas, we revealed its availability and reliability in mapping motor areas, demonstrating it as a promising tool in assisting neurosurgical practice and neuroscience research.

EAAI Journal 2024 Journal Article

A new approach to neural network via double hierarchy linguistic information: Application in robot selection

• Yang Zhang
• Saleem Abdullah
• Ihsan Ullah
• Fazal Ghani

Robotization is necessary to keep up with the constant changes in production, which calls for a staff with robotics expertise. A manufacturing business must also have the ability to swiftly change its production method. But today the procedure is drawn-out and complicated. In this study, inverse kinematics functionality and a machine learning model have been used to simulate an industrial robot’s movement in a digital environment. By using machine learning, less time and money must be invested in developing the procedure and determining the robot’s route. In this article, feed-forward double hierarchy linguistic neural networks with estimation information for double hierarchy linguistic term sets are proposed. First defined were the Yager operational rules and Yager aggregation operators for the double hierarchy linguistic terms set. Following that, we’ll discuss fuzzy neurons, feed-forward neural networks, simple neural networks, hybrid neural networks, and the sigmoid function. After that, explain feed-forward, double-hierarchy linguistic neural networks, including how their output is calculated. The weight vector of expert’s information is calculated by using the entropy measure with the help of Yager aggregation operators. Finally, we use the Yager t-norms to determine the output date of feed-forward double hierarchy linguistic neural networks and also find the output data. Linguistic neural network with Yager T-norms apply to the Robot selection for manufacturing bussing. The proposed approach of linguistic neural network are compared with Extended TOPSIS methods and GRA method for ranking.

AAAI Conference 2024 Conference Paper

Collaborative Weakly Supervised Video Correlation Learning for Procedure-Aware Instructional Video Analysis

• Tianyao He
• Huabin Liu
• Yuxi Li
• Xiao Ma
• Cheng Zhong
• Yang Zhang
• Weiyao Lin

Video Correlation Learning (VCL), which aims to analyze the relationships between videos, has been widely studied and applied in various general video tasks. However, applying VCL to instructional videos is still quite challenging due to their intrinsic procedural temporal structure. Specifically, procedural knowledge is critical for accurate correlation analyses on instructional videos. Nevertheless, current procedure-learning methods heavily rely on step-level annotations, which are costly and not scalable. To address this problem, we introduce a weakly supervised framework called Collaborative Procedure Alignment (CPA) for procedure-aware correlation learning on instructional videos. Our framework comprises two core modules: collaborative step mining and frame-to-step alignment. The collaborative step mining module enables simultaneous and consistent step segmentation for paired videos, leveraging the semantic and temporal similarity between frames. Based on the identified steps, the frame-to-step alignment module performs alignment between the frames and steps across videos. The alignment result serves as a measurement of the correlation distance between two videos. We instantiate our framework in two distinct instructional video tasks: sequence verification and action quality assessment. Extensive experiments validate the effectiveness of our approach in providing accurate and interpretable correlation analyses for instructional videos.

IJCAI Conference 2024 Conference Paper

Continual Compositional Zero-Shot Learning

• Yang Zhang
• Songhe Feng
• Jiazheng Yuan

Compositional Zero-Shot Learning (CZSL) aims to recognize unseen compositions with the knowledge learned from seen compositions, where each composition is composed of two primitives (attribute and object). However, existing CZSL methods are designed to learn compositions from fixed primitive set, which cannot handle the continually expanding primitive set in real-world applications. In this paper, we propose a new CZSL setting, named Continual Compositional Zero-Shot Learning (CCZSL), which requires the model to recognize unseen compositions composed of learned primitive set while continually increasing the size of learned primitive set. Contextuality and catastrophic forgetting are the main issues to be addressed in this setting. Specifically, we capture similar contextuality in compositions through several learnable Super-Primitives that can modify the invariant primitive embedding to better adapt the contextuality in the corresponding composition. Then we introduce a dual knowledge distillation loss which aims at maintaining old knowledge learned from previous sessions and avoiding overfitting of new session. We design the CCZSL evaluation protocol and conduct extensive experiments on widely used benchmarks, demonstrating the superiority of our method compared to the state-of-the-art CZSL methods.

ICML Conference 2024 Conference Paper

Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learning

• Xiaoyu Wen
• Chenjia Bai
• Kang Xu
• Xudong Yu
• Yang Zhang
• Xuelong Li 0001
• Zhen Wang 0004

Cross-domain offline reinforcement learning leverages source domain data with diverse transition dynamics to alleviate the data requirement for the target domain. However, simply merging the data of two domains leads to performance degradation due to the dynamics mismatch. Existing methods address this problem by measuring the dynamics gap via domain classifiers while relying on the assumptions of the transferability of paired domains. In this paper, we propose a novel representation-based approach to measure the domain gap, where the representation is learned through a contrastive objective by sampling transitions from different domains. We show that such an objective recovers the mutual-information gap of transition functions in two domains without suffering from the unbounded issue of the dynamics gap in handling significantly different domains. Based on the representations, we introduce a data filtering algorithm that selectively shares transitions from the source domain according to the contrastive score functions. Empirical results on various tasks demonstrate that our method achieves superior performance, using only 10% of the target data to achieve 89. 2% of the performance on 100% target dataset with state-of-the-art methods.

EAAI Journal 2024 Journal Article

Efficient segmentation with texture in ore images based on box-supervised approach

• Guodong Sun
• Delong Huang
• Yuting Peng
• Le Cheng
• Bo Wu
• Yang Zhang

Image segmentation methods have been utilized to determine the particle size distribution of crushed ores. Due to the complex working environment, high-powered computing equipment is difficult to deploy. At the same time, the ore distribution is stacked, and it is difficult to identify the complete features. To address this issue, an effective box-supervised technique with texture features is provided for ore image segmentation that can identify complete and independent ores. Firstly, a ghost feature pyramid network (Ghost-FPN) is proposed to process the features obtained from the backbone to reduce redundant semantic information and computation generated by complex networks. Then, an optimized detection head is proposed to obtain the feature to maintain accuracy. Finally, Lab color space (Lab) and local binary patterns (LBP) texture features are combined to form a fusion feature similarity-based loss function to improve accuracy while incurring no loss. Experiments on MS COCO have shown that the proposed fusion features are also worth studying on other types of datasets. Extensive experimental results demonstrate the effectiveness of the proposed method, which achieves over 50 frames per second with a small model size of 21. 6 MB. Meanwhile, the method maintains a high level of accuracy 67. 8 in A P 50 b o x and 47. 7 in A P 50 m a s k compared with the state-of-the-art approaches on ore image dataset, even better than bounding box tightness prior (BBTP) by 10. 4/1. 3 on A P 50 b o x / A P 50 m a s k metrics with the ResNet50 as backbone. The source code is available at https: //github. com/MVME-HBUT/OREINST.

EAAI Journal 2024 Journal Article

FS-OreDet: Feature enhancement and relationship exploration for boosting few-shot object detector of ore images

• Guodong Sun
• Le Cheng
• Jinyu Liu
• Yuting Peng
• Chengming Xu
• Yanwei Fu
• Bo Wu
• Yang Zhang

In the ore beneficiation process, large block detection is necessary to ensure production safety. This typically involves identifying oversized ore on the conveyor belt and preventing material blockage accidents in the transfer buffer bin between the ore feeding belt and the ore receiving belt. Methods based on deep learning can learn to construct complex features from a large amount of data, but they also require a large number of hand-made datasets for training. Although the existing few shot detection methods for ore images reduce the cost of manual labeling, the corresponding detection performance is insufficient. This article mainly explores how to improve the performance of the detector under the ore image detection task in the case of few labeled images. First, a shot enhancement block is proposed to enhance the valuable foreground information for higher-quality support features. Subsequently, we present a dual-attention region proposal network that effectively leverages support features to enhance the precision of generating candidate proposals. Finally, we propose a lightweight multi-relational detector to effectively evaluate the relationship between query and support proposals, leading to a substantial enhancement in guidance performance. The proposed few-shot object detector (FS-OreDet) achieves the best detection results with state-of-the-art methods with an average precision ( A P ) of 55. 1, a speed of 57 frames per second ( F P S ), and a model size of only 17 M B. Furthermore, our framework adeptly captures the feature information of ore images with substantial data. The detector’s accuracy achieves a significant improvement of 14% in A P. Compared with general object detectors, the performance of the detector ranks first and meets the requirements for outdoor scene deployment.

NeurIPS Conference 2024 Conference Paper

HairDiffusion: Vivid Multi-Colored Hair Editing via Latent Diffusion

• Yu Zeng
• Yang Zhang
• Jiachen Liu
• Linlin Shen
• Kaijun Deng
• Weizhao He
• Jinbao Wang

Hair editing is a critical image synthesis task that aims to edit hair color and hairstyle using text descriptions or reference images, while preserving irrelevant attributes (e. g. , identity, background, cloth). Many existing methods are based on StyleGAN to address this task. However, due to the limited spatial distribution of StyleGAN, it struggles with multiple hair color editing and facial preservation. Considering the advancements in diffusion models, we utilize Latent Diffusion Models (LDMs) for hairstyle editing. Our approach introduces Multi-stage Hairstyle Blend (MHB), effectively separating control of hair color and hairstyle in diffusion latent space. Additionally, we train a warping module to align the hair color with the target region. To further enhance multi-color hairstyle editing, we fine-tuned a CLIP model using a multi-color hairstyle dataset. Our method not only tackles the complexity of multi-color hairstyles but also addresses the challenge of preserving original colors during diffusion editing. Extensive experiments showcase the superiority of our method in editing multi-color hairstyles while preserving facial attributes given textual descriptions and reference images.

NeurIPS Conference 2024 Conference Paper

Learning Distinguishable Trajectory Representation with Contrastive Loss

• Tianxu Li
• Kun Zhu
• Juan Li
• Yang Zhang

Policy network parameter sharing is a commonly used technique in advanced deep multi-agent reinforcement learning (MARL) algorithms to improve learning efficiency by reducing the number of policy parameters and sharing experiences among agents. Nevertheless, agents that share the policy parameters tend to learn similar behaviors. To encourage multi-agent diversity, prior works typically maximize the mutual information between trajectories and agent identities using variational inference. However, this category of methods easily leads to inefficient exploration due to limited trajectory visitations. To resolve this limitation, inspired by the learning of pre-trained models, in this paper, we propose a novel Contrastive Trajectory Representation (CTR) method based on learning distinguishable trajectory representations to encourage multi-agent diversity. Specifically, CTR maps the trajectory of an agent into a latent trajectory representation space by an encoder and an autoregressive model. To achieve the distinguishability among trajectory representations of different agents, we introduce contrastive learning to maximize the mutual information between the trajectory representations and learnable identity representations of different agents. We implement CTR on top of QMIX and evaluate its performance in various cooperative multi-agent tasks. The empirical results demonstrate that our proposed CTR yields significant performance improvement over the state-of-the-art methods.

NeurIPS Conference 2024 Conference Paper

LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing

• Xiaonan Nie
• Qibin Liu
• Fangcheng Fu
• Shenhan Zhu
• Xupeng Miao
• Xiaoyang Li
• Yang Zhang
• Shouda Liu

Larger transformer models perform better on various downstream tasks but require more cost to scale up the model size. To efficiently enlarge models, the Mixture-of-Expert (MoE) architecture is widely adopted, which consists of a gate network and a series of experts and keep the training cost constant by routing the input data to a fixed number of experts instead of all. In existing large-scale MoE training systems, experts would be distributed among different GPUs for parallelization, and thus input data requires additional all-to-all communication to access the target expert and conduct corresponding computation. However, upon evaluating the training process of three mainstream MoE models on commonly used GPU clusters, we found that the all-to-all communication ratio averaged around 45\%, which significantly hinders the training efficiency and scalability of MoE models. In this paper, we propose LSH-MoE, a communication-efficient MoE training framework using locality-sensitive hashing (LSH). We first present the problems of scaling MoE training in existing systems and highlight the potential of exploiting token similarity to facilitate data compression. Then, we introduce an efficient LSH-based compression technique, which utilizes the cross-polytope hashing for rapid clustering and implements a residual-based error compensation scheme to alleviate the adverse impact of compression. To verify the effectiveness of our methods, we conduct experiments on both language models (e. g. , RoBERTa, GPT, and T5) and vision models (e. g. , Swin) for both pre-training and fine-tuning tasks. The results demonstrate that our method substantially outperforms its counterparts across different tasks by 1. 28-2. 2$\times$ of speedup.

NeurIPS Conference 2024 Conference Paper

MECD: Unlocking Multi-Event Causal Discovery in Video Reasoning

• Tieyuan Chen
• Huabin Liu
• Tianyao He
• Yihang Chen
• Chaofan Gan
• Xiao Ma
• Cheng Zhong
• Yang Zhang

Video causal reasoning aims to achieve a high-level understanding of video content from a causal perspective. However, current video reasoning tasks are limited in scope, primarily executed in a question-answering paradigm and focusing on short videos containing only a single event and simple causal relationships, lacking comprehensive and structured causality analysis for videos with multiple events. To fill this gap, we introduce a new task and dataset, Multi-Event Causal Discovery (MECD). It aims to uncover the causal relationships between events distributed chronologically across long videos. Given visual segments and textual descriptions of events, MECD requires identifying the causal associations between these events to derive a comprehensive, structured event-level video causal diagram explaining why and how the final result event occurred. To address MECD, we devise a novel framework inspired by the Granger Causality method, using an efficient mask-based event prediction model to perform an Event Granger Test, which estimates causality by comparing the predicted result event when premise events are masked versus unmasked. Furthermore, we integrate causal inference techniques such as front-door adjustment and counterfactual inference to address challenges in MECD like causality confounding and illusory causality. Experiments validate the effectiveness of our framework in providing causal relationships in multi-event videos, outperforming GPT-4o and VideoLLaVA by 5. 7% and 4. 1%, respectively.

NeurIPS Conference 2024 Conference Paper

Memory-Efficient Gradient Unrolling for Large-Scale Bi-level Optimization

• Qianli Shen
• Yezhen Wang
• Zhouhao Yang
• Xiang Li
• Haonan Wang
• Yang Zhang
• Jonathan Scarlett
• Zhanxing Zhu

Bi-level optimizaiton (BO) has become a fundamental mathematical framework for addressing hierarchical machine learning problems. As deep learning models continue to grow in size, the demand for scalable bi-level optimization has become increasingly critical. Traditional gradient-based bi-level optimizaiton algorithms, due to their inherent characteristics, are ill-suited to meet the demands of large-scale applications. In this paper, we introduce **F**orward **G**radient **U**nrolling with **F**orward **G**radient, abbreviated as **$($FG$)^2$U**, which achieves an unbiased stochastic approximation of the meta gradient for bi-level optimizaiton. $($FG$)^2$U circumvents the memory and approximation issues associated with classical bi-level optimizaiton approaches, and delivers significantly more accurate gradient estimates than existing large-scale bi-level optimizaiton approaches. Additionally, $($FG$)^2$U is inherently designed to support parallel computing, enabling it to effectively leverage large-scale distributed computing systems to achieve significant computational efficiency. In practice, $($FG$)^2$U and other methods can be strategically placed at different stages of the training process to achieve a more cost-effective two-phase paradigm. Further, $($FG$)^2$U is easy to implement within popular deep learning frameworks, and can be conveniently adapted to address more challenging zeroth-order bi-level optimizaiton scenarios. We provide a thorough convergence analysis and a comprehensive practical discussion for $($FG$)^2$U, complemented by extensive empirical evaluations, showcasing its superior performance in diverse large-scale bi-level optimizaiton tasks.

EAAI Journal 2024 Journal Article

Multiple prior representation learning for self-supervised monocular depth estimation via hybrid transformer

• Guodong Sun
• Junjie Liu
• Mingxuan Liu
• Moyun Liu
• Yang Zhang

Self-supervised monocular depth estimation aims to infer depth information without relying on labeled data. However, the lack of labeled information poses a significant challenge to the model’s representation, limiting its ability to capture the intricate details of the scene accurately. Prior information can potentially mitigate this issue, enhancing the model’s understanding of scene structure and texture. Nevertheless, solely relying on a single type of prior information often falls short when dealing with complex scenes, necessitating improvements in generalization performance. To address these challenges, we introduce a novel self-supervised monocular depth estimation model that leverages multiple priors to bolster representation capabilities across spatial, context, and semantic dimensions. Specifically, we employ a hybrid transformer and a lightweight pose network to obtain long-range spatial priors in the spatial dimension. Then, the context prior attention is designed to improve generalization, particularly in complex structures or untextured areas. In addition, semantic priors are introduced by leveraging semantic boundary loss, and semantic prior attention is supplemented, further refining the semantic features extracted by the decoder. Experiments on three diverse datasets demonstrate the effectiveness of the proposed model. It integrates multiple priors to comprehensively enhance the representation ability, improving the accuracy and reliability of depth estimation. Codes are available at: https: //github. com/MVME-HBUT/MPRLNet.

AAAI Conference 2024 Conference Paper

Polyper: Boundary Sensitive Polyp Segmentation

• Hao Shao
• Yang Zhang
• Qibin Hou

We present a new boundary sensitive framework for polyp segmentation, termed Polyper.Our method is motivated by a clinical approach that seasoned medical practitioners often leverage the inherent features of interior polyp regions to tackle blurred boundaries.Inspired by this, we propose to explicitly leverages boundary regions to bolster the model's boundary discrimination capability while minimizing computational resource wastage. Our approach first extracts low-confidence boundary regions and high-confidence prediction regions from an initial segmentation map through differentiable morphological operators.Then, we design the boundary sensitive attention that concentrates on augmenting the features near the boundary regions using the high-confidence prediction region's characteristics to generate good segmentation results.Our proposed method can be seamlessly integrated with classical encoder networks, like ResNet-50, MiT-B1, and Swin Transformer.To evaludate the effectiveness of Polyper, we conduct experiments on five publicly available challenging datasets, and receive state-of-the-art performance on all of them. Code is available at https://github.com/haoshao-nku/medical_seg.git.

NeurIPS Conference 2024 Conference Paper

ProgressGym: Alignment with a Millennium of Moral Progress

• Tianyi Qiu
• Yang Zhang
• Xuchuan Huang
• Jasmine X. Li
• Jiaming Ji
• Yaodong Yang

Frontier AI systems, including large language models (LLMs), hold increasing influence over the epistemology of human users. Such influence can reinforce prevailing societal values, potentially contributing to the lock-in of misguided moral beliefs and, consequently, the perpetuation of problematic moral practices on a broad scale. We introduce progress alignment as a technical solution to mitigate this imminent risk. Progress alignment algorithms learn to emulate the mechanics of human moral progress, thereby addressing the susceptibility of existing alignment methods to contemporary moral blindspots. To empower research in progress alignment, we introduce ProgressGym, an experimental framework allowing the learning of moral progress mechanics from history, in order to facilitate future progress in real-world moral decisions. Leveraging 9 centuries of historical text and 18 historical LLMs, ProgressGym enables codification of real-world progress alignment challenges into concrete benchmarks. Specifically, we introduce three core challenges: tracking evolving values (PG-Follow), preemptively anticipating moral progress (PG-Predict), and regulating the feedback loop between human and AI value shifts (PG-Coevolve). Alignment methods without a temporal dimension are inapplicable to these tasks. In response, we present lifelong and extrapolative algorithms as baseline methods of progress alignment, and build an open leaderboard soliciting novel algorithms and challenges.

NeurIPS Conference 2024 Conference Paper

Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference

• Jiabao Ji
• Yujian Liu
• Yang Zhang
• Gaowen Liu
• Ramana R. Kompella
• Sijia Liu
• Shiyu Chang

As Large Language Models (LLMs) demonstrate extensive capability in learning from documents, LLM unlearning becomes an increasingly important research area to address concerns of LLMs in terms of privacy, copyright, etc. A conventional LLM unlearning task typically involves two goals: (1) The target LLM should forget the knowledge in the specified forget documents; and (2) it should retain the other knowledge that the LLM possesses, for which we assume access to a small number of retain documents. To achieve both goals, a mainstream class of LLM unlearning methods introduces an optimization framework with a combination of two objectives – maximizing the prediction loss on the forget documents while minimizing that on the retain documents, which suffers from two challenges, degenerated output and catastrophic forgetting. In this paper, we propose a novel unlearning framework called Unlearning from Logit Difference (ULD), which introduces an assistant LLM that aims to achieve the opposite of the unlearning goals: remembering the forget documents and forgetting the retain knowledge. ULD then derives the unlearned LLM by computing the logit difference between the target and the assistant LLMs. We show that such reversed objectives would naturally resolve both aforementioned challenges while significantly improving the training efficiency. Extensive experiments demonstrate that our method efficiently achieves the intended forgetting while preserving the LLM’s overall capabilities, reducing training time by more than threefold. Notably, our method loses 0% of model utility on the ToFU benchmark, whereas baseline methods may sacrifice 17% of utility on average to achieve comparable forget quality.

ICML Conference 2024 Conference Paper

The Stronger the Diffusion Model, the Easier the Backdoor: Data Poisoning to Induce Copyright BreachesWithout Adjusting Finetuning Pipeline

• Haonan Wang
• Qianli Shen
• Yao Tong
• Yang Zhang
• Kenji Kawaguchi

The commercialization of text-to-image diffusion models (DMs) brings forth potential copyright concerns. Despite numerous attempts to protect DMs from copyright issues, the vulnerabilities of these solutions are underexplored. In this study, we formalized the Copyright Infringement Attack on generative AI models and proposed a backdoor attack method, SilentBadDiffusion, to induce copyright infringement without requiring access to or control over training processes. Our method strategically embeds connections between pieces of copyrighted information and text references in poisoning data while carefully dispersing that information, making the poisoning data inconspicuous when integrated into a clean dataset. Our experiments show the stealth and efficacy of the poisoning data. When given specific text prompts, DMs trained with a poisoning ratio of 0. 20% can produce copyrighted images. Additionally, the results reveal that the more sophisticated the DMs are, the easier the success of the attack becomes. These findings underline potential pitfalls in the prevailing copyright protection strategies and underscore the necessity for increased scrutiny to prevent the misuse of DMs.

JBHI Journal 2024 Journal Article

Time-Frequency-Space EEG Decoding Model Based on Dense Graph Convolutional Network for Stroke

• Jiancai Leng
• Han Li
• Weiyou Shi
• Licai Gao
• Chengyan Lv
• Chen Wang
• Fangzhou Xu
• Yang Zhang

Stroke, a sudden cerebrovascular ailment resulting from brain tissue damage, has prompted the use of motor imagery (MI)-based Brain-Computer Interface (BCI) systems in stroke rehabilitation. However, analyzing EEG signals from stroke patients is challenging because of their low signal-to-noise ratio and high variability. Therefore, we propose a novel approach that combines the modified S-transform (MST) and a dense graph convolutional network (DenseGCN) algorithm to enhance the MI-BCI performance across time, frequency, and space domains. MST is a time-frequency analysis method that efficiently concentrates energy in EEG signals, while DenseGCN is a deep learning model that uses EEG feature maps from each layer as inputs for subsequent layers, facilitating feature reuse and hyper-parameters optimization. Our approach outperforms conventional networks, achieving a peak classification accuracy of 90. 22% and an average information transfer rate (ITR) of 68. 52 bits per minute. Moreover, we conduct an in-depth analysis of the event-related desynchronization/event-related synchronization (ERD/ERS) phenomenon in the deep-level EEG features of stroke patients. Our experimental results confirm the feasibility and efficacy of the proposed approach for MI-BCI rehabilitation systems.

ICRA Conference 2024 Conference Paper

Unconstrained Model Predictive Control for Robot Navigation under Uncertainty

• Senthil Hariharan Arul
• Jong Jin Park
• Vishnu Prem
• Yang Zhang
• Dinesh Manocha

In this paper, we present a probabilistic and unconstrained model predictive control formulation for robot navigation under uncertainty. We present (1) a closed-form approximation of the probability of collision that naturally models the propagation of uncertainty over the planning horizon and is computationally cheap to evaluate, and (2) a collision-cost formulation which provably preserves forward invariance (i. e. , keeps the robot away from obstacles) when combined with the probability formulation. Notably, our formulation avoids hard constraints by construction, which in turn avoids abrupt transitions in robot behavior around the constraint boundaries ensuring graceful navigation. Further, we present proof for the forward invariance and the stability of the approach. We compare the efficacy of our method with the baseline [1], which the proposed approach builds on. We demonstrate that the approach results in confident and safe robot navigation in tight spaces by smoothly slowing down the robot in low survivability environments (e. g. , tight corridors), but also allows it to move away from obstacles safely when needed.

AAAI Conference 2023 Conference Paper

A Crowd-AI Collaborative Duo Relational Graph Learning Framework towards Social Impact Aware Photo Classification

• Yang Zhang
• Ziyi Kou
• Lanyu Shang
• Huimin Zeng
• Zhenrui Yue
• Dong Wang

In artificial intelligence (AI), negative social impact (NSI) represents the negative effect on the society as a result of mistakes conducted by AI agents. While the photo classification problem has been widely studied in the AI community, the NSI made by photo misclassification is largely ignored due to the lack of quantitative measurements of the NSI and effective approaches to reduce it. In this paper, we focus on an NSI-aware photo classification problem where the goal is to develop a novel crowd-AI collaborative learning framework that leverages online crowd workers to quantitatively estimate and effectively reduce the NSI of misclassified photos. Our problem is motivated by the limitations of current NSI-aware photo classification approaches that either 1) cannot accurately estimate NSI because they simply model NSI as the semantic difference between true and misclassified categories or 2) require costly human annotations to estimate NSI of pairwise class categories. To address such limitations, we develop SocialCrowd, a crowdsourcing-based NSI-aware photo classification framework that explicitly reduces the NSI of photo misclassification by designing a duo relational NSI-aware graph with the NSI estimated by online crowd workers. The evaluation results on two large-scale image datasets show that SocialCrowd not only reduces the NSI of photo misclassification but also improves the classification accuracy on both datasets.

JBHI Journal 2023 Journal Article

A Siamese Network With Node Convolution for Individualized Predictions Based on Connectivity Maps Extracted From Resting-State fMRI Data

• Le Xu
• Hao Ma
• Yun Guan
• Jiangcong Liu
• Huifang Huang
• Yang Zhang
• Lixia Tian

Deep learning has demonstrated great potential for objective diagnosis of neuropsychiatric disorders based on neuroimaging data, which includes the promising resting-state functional magnetic resonance imaging (RS-fMRI). However, the insufficient sample size has long been a bottleneck for deep model training for the purpose. In this study, we proposed a Siamese network with node convolution (SNNC) for individualized predictions based on RS-fMRI data. With the involvement of Siamese network, which uses sample pair (rather than a single sample) as input, the problem of insufficient sample size can largely be alleviated. To adapt to connectivity maps extracted from RS-fMRI data, we applied node convolution to each of the two branches of the Siamese network. For regression purposes, we replaced the contrastive loss in classic Siamese network with the mean square error loss and thus enabled Siamese network to quantitatively predict label differences. The label of a test sample can be predicted based on any of the training samples, by adding the label of the training sample to the predicted label difference between them. The final prediction for a test sample in this study was made by averaging the predictions based on each of the training samples. The performance of the proposed SNNC was evaluated with age and IQ predictions based on a public dataset (Cam-CAN). The results indicated that SNNC can make effective predictions even with a sample size of as small as 40, and SNNC achieved state-of-the-art accuracy among a variety of deep models and standard machine learning approaches.

TIST Journal 2023 Journal Article

Disease Simulation in Airport Scenario Based on Individual Mobility Model

• Zhenyu Han
• Siran Ma
• Changzheng Gao
• Erzhuo Shao
• Yulai Xie
• Yang Zhang
• Lu Geng
• Yong Li

As the rapid-spreading disease COVID-19 occupies the world, most governments adopt strict control policies to alleviate the impact of the virus. These policies successfully reduced the prevalence and delayed the epidemic peak, while they are also associated with high economic and social costs. To bridge the microscopic epidemic transmission patterns and control policies, simulation systems play an important role. In this work, we propose an agent-based disease simulator for indoor public spaces, which contribute to most of the transmission in cities. As an example, we study Guangzhou Baiyun International Airport, which is one of the most bustling aviation hubs in China. Specifically, we design a high-efficiency mobility generation module to reconstruct the individual trajectories considering both lingering behavior and crowd mobility, which greatly enhances the credibility of the simulated mobility and ensures real-time performance. Based on the individual trajectories, we propose a multi-path disease transmission module optimized for indoor public spaces, which includes three main transmission paths as close contact transmission, aerosol transmission, and object surface transmission. We design a novel convolution-based algorithm to mimic the diffusion process, which can leverage the high concurrent capability of the graphics processing unit to accelerate the simulation process. Leveraging our simulation paradigm, the effectiveness of common policy interventions can be quantitatively evaluated. For mobility interventions, we find that lingering control is the most effective mobility intervention with 32.35% fewer infections, while increasing social distance and increasing walking speed have a similar effect with 15.15% and 18.02% fewer infections. It demonstrates the importance of introducing crowd mobility into disease transmission simulation. For transmission processes, we find the aerosol transmission involves in 99.99% of transmission, which highlights the importance of ventilation in indoor public spaces. Our simulation also demonstrates that without strict entrance detection to identify the input infections, only performing frequent disinfection cannot achieve desirable epidemic outcomes. Based on our simulation paradigm, we can shed light on better policy designs that achieve a good balance between disease spreading control and social costs.

YNIMG Journal 2023 Journal Article

Dual interaction between heartbeat-evoked responses and stimuli

• Yihui Zhang
• Jianfeng Zhang
• Musi Xie
• Nai Ding
• Yang Zhang
• Pengmin Qin

Heartbeat-evoked responses (HERs) can interact with external stimuli and play a crucial role in shaping perception, self-related processes, and emotional processes. On the one hand, the external stimulus could modulate HERs. On the other hand, the HERs could affect cognitive processing of the external stimulus. Whether the same neural mechanism underlies these two processes, however, remains unclear. Here, we investigated this interactive mechanism by measuring HERs using magnetoencephalography (MEG) and two name perception tasks. Specifically, we tested (1) how hearing a subject's own name (SON) modulates HERs and (2) how the judgment of an SON is biased by prestimulus HERs. The results showed a dual interaction between HERs and SON. In particular, SON can modulate HERs for heartbeats occurring from 200 to 1200 ms after SON presentation. In addition, prestimulus HERs can bias the SON judgment when a stimulus is presented. Importantly, MEG activities from these two types of interactions differed in spatial and temporal patterns, suggesting that they may be associated with distinct neural pathways. These findings extend our understanding of brain-heart interactions.

AAAI Conference 2023 Conference Paper

Dual Memory Aggregation Network for Event-Based Object Detection with Learnable Representation

• Dongsheng Wang
• Xu Jia
• Yang Zhang
• Xinyu Zhang
• Yaoyuan Wang
• Ziyang Zhang
• Dong Wang
• Huchuan Lu

Event-based cameras are bio-inspired sensors that capture brightness change of every pixel in an asynchronous manner. Compared with frame-based sensors, event cameras have microsecond-level latency and high dynamic range, hence showing great potential for object detection under high-speed motion and poor illumination conditions. Due to sparsity and asynchronism nature with event streams, most of existing approaches resort to hand-crafted methods to convert event data into 2D grid representation. However, they are sub-optimal in aggregating information from event stream for object detection. In this work, we propose to learn an event representation optimized for event-based object detection. Specifically, event streams are divided into grids in the x-y-t coordinates for both positive and negative polarity, producing a set of pillars as 3D tensor representation. To fully exploit information with event streams to detect objects, a dual-memory aggregation network (DMANet) is proposed to leverage both long and short memory along event streams to aggregate effective information for object detection. Long memory is encoded in the hidden state of adaptive convLSTMs while short memory is modeled by computing spatial-temporal correlation between event pillars at neighboring time intervals. Extensive experiments on the recently released event-based automotive detection dataset demonstrate the effectiveness of the proposed method.

ECAI Conference 2023 Conference Paper

FATRER: Full-Attention Topic Regularizer for Accurate and Robust Conversational Emotion Recognition

• Yuzhao Mao
• Di Lu
• Yang Zhang
• Xiaojie Wang 0006

This paper concentrates on the understanding of interlocutors’ emotions evoked in conversational utterances. Previous studies in this literature mainly focus on more accurate emotional predictions, while ignoring model robustness when the local context is corrupted by adversarial attacks. To maintain robustness while ensuring accuracy, we propose an emotion recognizer augmented by a full-attention topic regularizer, which enables an emotion-related global view when modeling the local context in a conversation. A joint topic modeling strategy is introduced to implement regularization from both representation and loss perspectives. To avoid over-regularization, we drop the constraints on prior distributions that exist in traditional topic modeling and perform probabilistic approximations based entirely on attention alignment. Experiments show that our models obtain more favorable results than state-of-the-art models, and gain convincing robustness under three types of adversarial attacks. Code: https: //github. com/ludybupt/FATRER.

TIST Journal 2023 Journal Article

Interior Individual Trajectory Simulation with Population Distribution Constraint

• Erzhuo Shao
• Zhenyu Han
• Yulai Xie
• Yang Zhang
• Lu Geng
• Yong Li

Individual trajectory generation plays an important role in simulation tasks, reconstructing fine-grained mobility behaviors that can be used to evaluate epidemic risks, congestion risks, or commercial profit. Previous research works adopt the Newton’s mechanic-based particle model as their core algorithm, such as the Social Force model. However, real-world human mobility behaviors hardly follow the particle models, especially in the interior scenes where interactions between pedestrians and environments matter. In this article, we propose a Social Force-based trajectory simulator for interior scenarios that improve both trajectory quality and generation speed for interior scenarios. First, we introduce prior scene knowledge to guide the generation process, where pedestrians are armed with exploration behaviors that follow the group-level distribution. It provides more flexibility to simulate complicated human behaviors rather than straight-line movements, generating high-quality individual trajectories. Experiments show that the correlation between the aggregated population distribution of generated trajectories and ground-truth distribution is improved by 11.84% by our method. Second, we optimize the algorithm procedure by introducing a caching mechanism for tenderized intermediate values, along with graph-processing-unit-based implementation. Compared with the baseline Social Force model, we reduced the time consumption by 95%. More importantly, based on our simulation paradigm, we quantitatively evaluate several common mobility interventions in our simulation scenario, which can shed light on better policy designs in public spaces.

IJCAI Conference 2023 Conference Paper

Long-term Wind Power Forecasting with Hierarchical Spatial-Temporal Transformer

• Yang Zhang
• Lingbo Liu
• Xinyu Xiong
• Guanbin Li
• Guoli Wang
• Liang Lin

Wind power is attracting increasing attention around the world due to its renewable, pollution-free, and other advantages. However, safely and stably integrating the high permeability intermittent power energy into electric power systems remains challenging. Accurate wind power forecasting (WPF) can effectively reduce power fluctuations in power system operations. Existing methods are mainly designed for short-term predictions and lack effective spatial-temporal feature augmentation. In this work, we propose a novel end-to-end wind power forecasting model named Hierarchical Spatial-Temporal Transformer Network (HSTTN) to address the long-term WPF problems. Specifically, we construct an hourglass-shaped encoder-decoder framework with skip-connections to jointly model representations aggregated in hierarchical temporal scales, which benefits long-term forecasting. Based on this framework, we capture the inter-scale long-range temporal dependencies and global spatial correlations with two parallel Transformer skeletons and strengthen the intra-scale connections with downsampling and upsampling operations. Moreover, the complementary information from spatial and temporal features is fused and propagated in each other via Contextual Fusion Blocks (CFBs) to promote the prediction further. Extensive experimental results on two large-scale real-world datasets demonstrate the superior performance of our HSTTN over existing solutions.

EAAI Journal 2023 Journal Article

Neural network-based analytical solver for Fokker–Planck equation

• Yang Zhang
• Run-Fa Zhang
• Ka-Veng Yuen

The Fokker–Planck equation has significant applications in dynamical systems. In recent years, some neural network methods have been used in combination with physical models to obtain its numerical solutions. However, it is also appealing if the analytical solution of the physical model can be obtained. This paper proposes a neural network-based method for the analytical solution of the FP equation. It relies on neural networks and uses their explicit model as the trial function for the FP equation. The trial function contains the weights and biases in the neural network. Therefore, the solving of the FP equation is converted into the calculation of the weights and biases. In the proposed method, the FP equations are first reduced to a set of easily solvable nonlinear algebraic equations using some trial functions, and then the corresponding weights and biases are determined using the method of pending coefficients. In this paper, linear and nonlinear numerical examples were used to verify the effectiveness of the proposed method. The results demonstrated that the proposed method can obtain the exact solution of the FP equations without data samples. Finally, the proposed method is compared in detail with physics-informed neural networks in terms of computational theory and computational effectiveness.

IJCAI Conference 2023 Conference Paper

On Adversarial Robustness of Demographic Fairness in Face Attribute Recognition

• Huimin Zeng
• Zhenrui Yue
• Lanyu Shang
• Yang Zhang
• Dong Wang

Demographic fairness has become a critical objective when developing modern visual models for identity-sensitive applications, such as face attribute recognition (FAR). While great efforts have been made to improve the fairness of the models, the investigation on the adversarial robustness of the fairness (e. g. , whether the fairness of the models could still be maintained under potential malicious fairness attacks) is largely ignored. Therefore, this paper explores the adversarial robustness of demographic fairness in FAR applications from both attacking and defending perspectives. In particular, we firstly present a novel fairness attack, who aims at corrupting the demographic fairness of face attribute classifiers. Next, to mitigate the effect of the fairness attack, we design an efficient defense algorithm called robust-fair training. With this defense, face attribute classifiers learn how to combat the bias introduced by the fairness attack. As such, the face attribute classifiers are not only trained to be fair, but the fairness is also robust. Our extensive experimental results show the effectiveness of both our proposed attack and defense methods across various model architectures and FAR applications. We believe our work could be strong baselines for future work on robust-fair AI models.

IJCAI Conference 2023 Conference Paper

On Optimizing Model Generality in AI-based Disaster Damage Assessment: A Subjective Logic-driven Crowd-AI Hybrid Learning Approach

• Yang Zhang
• Ruohan Zong
• Lanyu Shang
• Huimin Zeng
• Zhenrui Yue
• Na Wei
• Dong Wang

This paper focuses on the AI-based damage assessment (ADA) applications that leverage state-of-the-art AI techniques to automatically assess the disaster damage severity using online social media imagery data, which aligns well with the ''disaster risk reduction'' target under United Nations' Sustainable Development Goals (UN SDGs). This paper studies an ADA model generality problem where the objective is to address the limitation of current ADA solutions that are often optimized only for a single disaster event and lack the generality to provide accurate performance across different disaster events. To address this limitation, we work with domain experts and local community stakeholders in disaster response to develop CollabGeneral, a subjective logic-driven crowd-AI collaborative learning framework that integrates AI and crowdsourced human intelligence into a principled learning framework to address the ADA model generality problem. Extensive experiments on four real-world ADA datasets demonstrate that CollabGeneral consistently outperforms the state-of-the-art baselines by significantly improving the ADA model generality across different disasters.

AAAI Conference 2023 Conference Paper

Pseudo Label-Guided Model Inversion Attack via Conditional Generative Adversarial Network

• Xiaojian Yuan
• Kejiang Chen
• Jie Zhang
• Weiming Zhang
• Nenghai Yu
• Yang Zhang

Model inversion (MI) attacks have raised increasing concerns about privacy, which can reconstruct training data from public models. Indeed, MI attacks can be formalized as an optimization problem that seeks private data in a certain space. Recent MI attacks leverage a generative adversarial network (GAN) as an image prior to narrow the search space, and can successfully reconstruct even the high-dimensional data (e.g., face images). However, these generative MI attacks do not fully exploit the potential capabilities of the target model, still leading to a vague and coupled search space, i.e., different classes of images are coupled in the search space. Besides, the widely used cross-entropy loss in these attacks suffers from gradient vanishing. To address these problems, we propose Pseudo Label-Guided MI (PLG-MI) attack via conditional GAN (cGAN). At first, a top-n selection strategy is proposed to provide pseudo-labels for public data, and use pseudo-labels to guide the training of the cGAN. In this way, the search space is decoupled for different classes of images. Then a max-margin loss is introduced to improve the search process on the subspace of a target class. Extensive experiments demonstrate that our PLG-MI attack significantly improves the attack success rate and visual quality for various datasets and models, notably, 2 ∼ 3× better than state-of-the-art attacks under large distributional shifts. Our code is available at: https://github.com/LetheSec/PLG-MI-Attack.

EAAI Journal 2022 Journal Article

A global interactive attention-based lightweight denoising network for locating internal defects of CFRP laminates

• Bo Yang
• Yang Zhang
• Shilong Wang
• Weichun Xu
• Meng Xiao
• Yan He
• Fan Mo

Carbon fiber reinforced plastic (CFRP) has become one of the main structural materials for aerospace vehicles. However, some internal defects are prone to occur and have potential to cause significant losses of life and property. Currently, the detection of internal defects for CFRP mainly relies on ultrasonic, and other technologies, while they have disadvantages of low efficiency, and poor adaptability. Therefore, this paper explores a novel method to locate internal defects of CFRP laminates by analyzing vibration signals. Firstly, a signal acquisition scheme is designed. Then, a global interactive attention-based lightweight denoising network (GIALDN) is designed to analyze vibration signals and locate internal defects of CFRP laminates. In GIALDN, the threshold denoising method is used to eliminate noise-related features and improve feature discrimination; a global interactive attention module is designed, which makes the network pay more attention to the valid features while realizing the global interactive connection and obtains the rich contextual features; combining with the convolution layer of de-pooling strategy and multi-layer convolution using the residual connection, the backbone of the network is formed. Finally, an experimental platform is established to test the performance of GIALDN. Results show that the location accuracy of GIALDN can reach 98. 68%, which is more than 15% higher than those of VGGnet11 and FaultNet, and is also superior to those of LSTM, RNN, Rsenet18, SEresnet18 and Densenet121. Lastly, the location accuracies of GIALDN on CFRP laminates with the same thickness and different stacking sequences are investigated and a good model applicability can be observed.

NeurIPS Conference 2022 Conference Paper

Amplifying Membership Exposure via Data Poisoning

• Yufei Chen
• Chao Shen
• Yun Shen
• Cong Wang
• Yang Zhang

As in-the-wild data are increasingly involved in the training stage, machine learning applications become more susceptible to data poisoning attacks. Such attacks typically lead to test-time accuracy degradation or controlled misprediction. In this paper, we investigate the third type of exploitation of data poisoning - increasing the risks of privacy leakage of benign training samples. To this end, we demonstrate a set of data poisoning attacks to amplify the membership exposure of the targeted class. We first propose a generic dirty-label attack for supervised classification algorithms. We then propose an optimization-based clean-label attack in the transfer learning scenario, whereby the poisoning samples are correctly labeled and look "natural" to evade human moderation. We extensively evaluate our attacks on computer vision benchmarks. Our results show that the proposed attacks can substantially increase the membership inference precision with minimum overall test-time model performance degradation. To mitigate the potential negative impacts of our attacks, we also investigate feasible countermeasures.

AAAI Conference 2022 Conference Paper

An Adversarial Framework for Generating Unseen Images by Activation Maximization

• Yang Zhang
• Wang Zhou
• Gaoyuan Zhang
• David Cox
• Shiyu Chang

Activation maximization (AM) refers to the task of generating input examples that maximize the activation of a target class of a classifier, which can be used for class-conditional image generation and model interpretation. A popular class of AM method, GAN-based AM, introduces a GAN pre-trained on a large image set, and performs AM over its input random seed or style embeddings, so that the generated images are natural and adversarial attacks are prevented. Most of these methods would require the image set to contain some images of the target class to be visualized. Otherwise they tend to generate other seen class images that most maximizes the target class activation. In this paper, we aim to tackle the case where information about the target class is completely removed from the image set. This would ensure that the generated images truly reflect the target class information residing in the classifier, not the target class information in the image set, which contributes to a more faithful interpretation technique. To this end, we propose PROBE- GAN, a GAN-based AM algorithm capable of generating image classes unseen in the image set. Rather than using a pre-trained GAN, PROBEGAN trains a new GAN with AM explicitly included in its training objective. PROBEGAN consists of a class-conditional generator, a seen-class discriminator, and an all-class unconditional discriminator. It can be shown that such a framework can generate images with the features of the unseen target class, while retaining the naturalness as depicted in the image set. Experiments have shown that PROBEGAN can generate unseen-class images with much higher quality than the baselines. We also explore using PROBEGAN as a model interpretation tool. Our code is at https: //github. com/csmiler/ProbeGAN/.

IJCAI Conference 2022 Conference Paper

BiFSMN: Binary Neural Network for Keyword Spotting

• Haotong Qin
• Xudong Ma
• Yifu Ding
• Xiaoyang Li
• Yang Zhang
• Yao Tian
• Zejun Ma
• Jie Luo

The deep neural networks, such as the Deep-FSMN, have been widely studied for keyword spotting (KWS) applications. However, computational resources for these networks are significantly constrained since they usually run on-call on edge devices. In this paper, we present BiFSMN, an accurate and extreme-efficient binary neural network for KWS. We first construct a High-frequency Enhancement Distillation scheme for the binarization-aware training, which emphasizes the high-frequency information from the full-precision network's representation that is more crucial for the optimization of the binarized network. Then, to allow the instant and adaptive accuracy-efficiency trade-offs at runtime, we also propose a Thinnable Binarization Architecture to further liberate the acceleration potential of the binarized network from the topology perspective. Moreover, we implement a Fast Bitwise Computation Kernel for BiFSMN on ARMv8 devices which fully utilizes registers and increases instruction throughput to push the limit of deployment efficiency. Extensive experiments show that BiFSMN outperforms existing binarization methods by convincing margins on various datasets and is even comparable with the full-precision counterpart (e. g. , less than 3% drop on Speech Commands V1-12). We highlight that benefiting from the thinnable architecture and the optimized 1-bit implementation, BiFSMN can achieve an impressive 22. 3x speedup and 15. 5x storage-saving on real-world edge hardware.

IJCAI Conference 2022 Conference Paper

Crowd, Expert & AI: A Human-AI Interactive Approach Towards Natural Language Explanation Based COVID-19 Misinformation Detection

• Ziyi Kou
• Lanyu Shang
• Yang Zhang
• Zhenrui Yue
• Huimin Zeng
• Dong Wang

In this paper, we study an explainable COVID-19 misinformation detection problem where the goal is to accurately identify COVID-19 misleading posts on social media and explain the posts with natural language explanations (NLEs). Our problem is motivated by the limitations of current explainable misinformation detection approaches that cannot provide NLEs for COVID-19 posts due to the lack of sufficient professional COVID-19 knowledge for supervision. To address such a limitation, we develop CEA-COVID, a crowd-expert-AI framework that jointly exploits the common logical reasoning ability of online crowd workers and the professional knowledge of COVID-19 experts to effectively generate NLEs for detecting and explaining COVID-19 misinformation. We evaluate CEA-COVID using two public COVID-19 misinformation datasets on social media. Results demonstrate that CEA-COVID outperforms existing explainable misinformation detection models in terms of both explainability and detection accuracy.

NeurIPS Conference 2022 Conference Paper

Fairness Reprogramming

• Guanhua Zhang
• Yihua Zhang
• Yang Zhang
• Wenqi Fan
• Qing Li
• Sijia Liu
• Shiyu Chang

Despite a surge of recent advances in promoting machine Learning (ML) fairness, the existing mainstream approaches mostly require training or finetuning the entire weights of the neural network to meet the fairness criteria. However, this is often infeasible in practice for those large-scale trained models due to large computational and storage costs, low data efficiency, and model privacy issues. In this paper, we propose a new generic fairness learning paradigm, called FairReprogram, which incorporates the model reprogramming technique. Specifically, FairReprogram considers the case where models can not be changed and appends to the input a set of perturbations, called the fairness trigger, which is tuned towards the fairness criteria under a min-max formulation. We further introduce an information-theoretic framework that explains why and under what conditions fairness goals can be achieved using the fairness trigger. We show both theoretically and empirically that the fairness trigger can effectively obscure demographic biases in the output prediction of fixed ML models by providing false demographic information that hinders the model from utilizing the correct demographic information to make the prediction. Extensive experiments on both NLP and CV datasets demonstrate that our method can achieve better fairness improvements than retraining-based methods with far less data dependency under two widely-used fairness criteria. Codes are available at https: //github. com/UCSB-NLP-Chang/Fairness-Reprogramming. git.

TIST Journal 2022 Journal Article

FairSR: Fairness-aware Sequential Recommendation through Multi-Task Learning with Preference Graph Embeddings

• Cheng-Te Li
• Cheng Hsu
• Yang Zhang

Sequential recommendation (SR) learns from the temporal dynamics of user-item interactions to predict the next ones. Fairness-aware recommendation mitigates a variety of algorithmic biases in the learning of user preferences. This article aims at bringing a marriage between SR and algorithmic fairness. We propose a novel fairness-aware sequential recommendation task, in which a new metric, interaction fairness, is defined to estimate how recommended items are fairly interacted by users with different protected attribute groups. We propose a multi-task learning-based deep end-to-end model, FairSR, which consists of two parts. One is to learn and distill personalized sequential features from the given user and her item sequence for SR. The other is fairness-aware preference graph embedding (FPGE). The aim of FPGE is two-fold: incorporating the knowledge of users’ and items’ attributes and their correlation into entity representations, and alleviating the unfair distributions of user attributes on items. Extensive experiments conducted on three datasets show FairSR can outperform state-of-the-art SR models in recommendation performance. In addition, the recommended items by FairSR also exhibit promising interaction fairness.

NeurIPS Conference 2022 Conference Paper

Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processing

• Yonggan Fu
• Yang Zhang
• Kaizhi Qian
• Zhifan Ye
• Zhongzhi Yu
• Cheng-I Jeff Lai
• Celine Lin

Self-supervised learning (SSL) for rich speech representations has achieved empirical success in low-resource Automatic Speech Recognition (ASR) and other speech processing tasks, which can mitigate the necessity of a large amount of transcribed speech and thus has driven a growing demand for on-device ASR and other speech processing. However, advanced speech SSL models have become increasingly large, which contradicts the limited on-device resources. This gap could be more severe in multilingual/multitask scenarios requiring simultaneously recognizing multiple languages or executing multiple speech processing tasks. Additionally, strongly overparameterized speech SSL models tend to suffer from overfitting when being finetuned on low-resource speech corpus. This work aims to enhance the practical usage of speech SSL models towards a win-win in both enhanced efficiency and alleviated overfitting via our proposed S$^3$-Router framework, which for the first time discovers that simply discarding no more than 10% of model weights via only finetuning model connections of speech SSL models can achieve better accuracy over standard weight finetuning on downstream speech processing tasks. More importantly, S$^3$-Router can serve as an all-in-one technique to enable (1) a new finetuning scheme, (2) an efficient multilingual/multitask solution, (3) a state-of-the-art pruning technique, and (4) a new tool to quantitatively analyze the learned speech representation. We believe S$^3$-Router has provided a new perspective for practical deployment of speech SSL models. Our codes are available at: https: //github. com/GATECH-EIC/S3-Router.

YNIMG Journal 2022 Journal Article

Neural correlates of perceived emotions in human insula and amygdala for auditory emotion recognition

• Yang Zhang
• Wenjing Zhou
• Juan Huang
• Bo Hong
• Xiaoqin Wang

The emotional status of a speaker is an important non-linguistic cue carried by human voice and can be perceived by a listener in vocal communication. Understanding the neural circuits involved in processing emotions carried by human voice is crucial for understanding the neural basis of social interaction. Previous studies have shown that human insula and amygdala responded more selectively to emotional sounds than non-emotional sounds. However, it is not clear whether the neural selectivity to emotional sounds in these brain structures is determined by the emotion presented by a speaker which is associated with the acoustic properties of the sounds or by the emotion perceived by a listener. In this study, we recorded intracranial electroencephalography (iEEG) responses to emotional human voices while subjects performed emotion recognition tasks. We found that the iEEG responses of Heschl's gyrus (HG) and posterior insula were determined by the presented emotion, whereas the iEEG responses of anterior insula and amygdala were driven by the perceived emotion. These results suggest that the anterior insula and amygdala play a crucial role in conscious perception of emotions carried by human voice.

IJCAI Conference 2022 Conference Paper

On Attacking Out-Domain Uncertainty Estimation in Deep Neural Networks

• Huimin Zeng
• Zhenrui Yue
• Yang Zhang
• Ziyi Kou
• Lanyu Shang
• Dong Wang

In many applications with real-world consequences, it is crucial to develop reliable uncertainty estimation for the predictions made by the AI decision systems. Targeting at the goal of estimating uncertainty, various deep neural network (DNN) based uncertainty estimation algorithms have been proposed. However, the robustness of the uncertainty returned by these algorithms has not been systematically explored. In this work, to raise the awareness of the research community on robust uncertainty estimation, we show that state-of-the-art uncertainty estimation algorithms could fail catastrophically under our proposed adversarial attack despite their impressive performance on uncertainty estimation. In particular, we aim at attacking out-domain uncertainty estimation: under our attack, the uncertainty model would be fooled to make high-confident predictions for the out-domain data, which they originally would have rejected. Extensive experimental results on various benchmark image datasets show that the uncertainty estimated by state-of-the-art methods could be easily corrupted by our attack.

NeurIPS Conference 2021 Conference Paper

BCORLE($\lambda$): An Offline Reinforcement Learning and Evaluation Framework for Coupons Allocation in E-commerce Market

• Yang Zhang
• Bo Tang
• Qingyu Yang
• Dou An
• Hongyin Tang
• Chenyang Xi
• Xueying Li
• Feiyu Xiong

Coupons allocation is an important tool for enterprises to increase the activity and loyalty of users on the e-commerce market. One fundamental problem related is how to allocate coupons within a fixed budget while maximizing users' retention on the e-commerce platform. The online e-commerce environment is complicated and ever changing, so it requires the coupons allocation policy learning can quickly adapt to the changes of the company's business strategy. Unfortunately, existing studies with a huge computation overhead can hardly satisfy the requirements of real-time and fast-response in the real world. Specifically, the problem of coupons allocation within a fixed budget is usually formulated as a Lagrangian problem. Existing solutions need to re-learn the policy once the value of Lagrangian multiplier variable $\lambda$ is updated, causing a great computation overhead. Besides, a mature e-commerce market often faces tens of millions of users and dozens of types of coupons which construct the huge policy space, further increasing the difficulty of solving the problem. To tackle with above problems, we propose a budget constrained offline reinforcement learning and evaluation with $\lambda$-generalization (BCORLE($\lambda$)) framework. The proposed method can help enterprises develop a coupons allocation policy which greatly improves users' retention rate on the platform while ensuring the cost does not exceed the budget. Specifically, $\lambda$-generalization method is proposed to lead the policy learning process can be executed according to different $\lambda$ values adaptively, avoiding re-learning new polices from scratch. Thus the computation overhead is greatly reduced. Further, a novel offline reinforcement learning method and an off-policy evaluation algorithm are proposed for policy learning and policy evaluation, respectively. Finally, experiments on the simulation platform and real-world e-commerce market validate the effectiveness of our approach.

AAAI Conference 2021 Conference Paper

Coordination Between Individual Agents in Multi-Agent Reinforcement Learning

• Yang Zhang
• Qingyu Yang
• Dou An
• Chengwei Zhang

The existing multi-agent reinforcement learning methods (MARL) for determining the coordination between agents focus on either global-level or neighborhood-level coordination between agents. However the problem of coordination between individual agents is remain to be solved. It is crucial for learning an optimal coordinated policy in unknown multi-agent environments to analyze the agent’s roles and the correlation between individual agents. To this end, in this paper we propose an agent-level coordination based MARL method. Specifically, it includes two parts in our method. The first is correlation analysis between individual agents based on the Pearson, Spearman, and Kendall correlation coefficients; And the second is an agent-level coordinated training framework where the communication message between weakly correlated agents is dropped out, and a correlation based reward function is built. The proposed method is verified in four mixed cooperative-competitive environments. The experimental results show that the proposed method outperforms the state-of-the-art MARL methods and can measure the correlation between individual agents accurately.

NeurIPS Conference 2021 Conference Paper

Drawing Robust Scratch Tickets: Subnetworks with Inborn Robustness Are Found within Randomly Initialized Networks

• Yonggan Fu
• Qixuan Yu
• Yang Zhang
• Shang Wu
• Xu Ouyang
• David Cox
• Yingyan Lin

Deep Neural Networks (DNNs) are known to be vulnerable to adversarial attacks, i. e. , an imperceptible perturbation to the input can mislead DNNs trained on clean images into making erroneous predictions. To tackle this, adversarial training is currently the most effective defense method, by augmenting the training set with adversarial samples generated on the fly. \textbf{Interestingly, we discover for the first time that there exist subnetworks with inborn robustness, matching or surpassing the robust accuracy of the adversarially trained networks with comparable model sizes, within randomly initialized networks without any model training}, indicating that adversarial training on model weights is not indispensable towards adversarial robustness. We name such subnetworks Robust Scratch Tickets (RSTs), which are also by nature efficient. Distinct from the popular lottery ticket hypothesis, neither the original dense networks nor the identified RSTs need to be trained. To validate and understand this fascinating finding, we further conduct extensive experiments to study the existence and properties of RSTs under different models, datasets, sparsity patterns, and attacks, drawing insights regarding the relationship between DNNs’ robustness and their initialization/overparameterization. Furthermore, we identify the poor adversarial transferability between RSTs of different sparsity ratios drawn from the same randomly initialized dense network, and propose a Random RST Switch (R2S) technique, which randomly switches between different RSTs, as a novel defense method built on top of RSTs. We believe our findings about RSTs have opened up a new perspective to study model robustness and extend the lottery ticket hypothesis.

NeurIPS Conference 2021 Conference Paper

Fine-Grained Neural Network Explanation by Identifying Input Features with Predictive Information

• Yang Zhang
• Ashkan Khakzar
• Yawei Li
• Azade Farshad
• Seong Tae Kim
• Nassir Navab

One principal approach for illuminating a black-box neural network is feature attribution, i. e. identifying the importance of input features for the network’s prediction. The predictive information of features is recently proposed as a proxy for the measure of their importance. So far, the predictive information is only identified for latent features by placing an information bottleneck within the network. We propose a method to identify features with predictive information in the input domain. The method results in fine-grained identification of input features' information and is agnostic to network architecture. The core idea of our method is leveraging a bottleneck on the input that only lets input features associated with predictive latent features pass through. We compare our method with several feature attribution methods using mainstream feature attribution evaluation experiments. The code is publicly available.

EAAI Journal 2021 Journal Article

MLANet: Multi-Layer Anchor-free Network for generic lesion detection

• Zhe Liu
• Xi Xie
• Yuqing Song
• Yang Zhang
• Xuesheng Liu
• Jiawen Zhang
• Victor S. Sheng

In medical image processing, detecting lesions from computed tomography (CT) scans becomes an important research problem with increasing attention. However, this problem is nontrivial because lesions from different organs and parts reflect different characteristics as well as different sizes. Most conventional methods only use a single-scale architecture to detect lesion areas. To get rid of the drawbacks above in medical imaging, a multi-scale framework called MLANet is proposed. To deal with the scale imbalance problem, we design a new backbone—a mixed hourglass network, in which each hourglass module share different input sizes and orders to extract features from different scales. And then the information is sent to the proposed Strengthen Weighted Feature Pyramid Network (SWFPN), a multi-layer weighted feature fusion module, to combine more semantic and spatial information, especially for the case where the number of layers is small. Finally, a Center-to-Corner (C2C) transformation is proposed to deal with the inaccurate size prediction of lesions. It is a non-linear transformation function, aiming to make the predictions more stable and accurate. MLANet is an end-to-end network and is easy to train. In our experiment, it achieves 65. 2% AP50, as well as 88. 3% in the sensitivity of FPs@4. 0 on the DeepLesion dataset, which exceeds many state-of-the-art detectors.

NeurIPS Conference 2021 Conference Paper

PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition

• Cheng-I Jeff Lai
• Yang Zhang
• Alexander H. Liu
• Shiyu Chang
• Yi-Lun Liao
• Yung-Sung Chuang
• Kaizhi Qian
• Sameer Khurana

Self-supervised speech representation learning (speech SSL) has demonstrated the benefit of scale in learning rich representations for Automatic Speech Recognition (ASR) with limited paired data, such as wav2vec 2. 0. We investigate the existence of sparse subnetworks in pre-trained speech SSL models that achieve even better low-resource ASR results. However, directly applying widely adopted pruning methods such as the Lottery Ticket Hypothesis (LTH) is suboptimal in the computational cost needed. Moreover, we show that the discovered subnetworks yield minimal performance gain compared to the original dense network. We present Prune-Adjust-Re-Prune (PARP), which discovers and finetunes subnetworks for much better performance, while only requiring a single downstream ASR finetuning run. PARP is inspired by our surprising observation that subnetworks pruned for pre-training tasks need merely a slight adjustment to achieve a sizeable performance boost in downstream ASR tasks. Extensive experiments on low-resource ASR verify (1) sparse subnetworks exist in mono-lingual/multi-lingual pre-trained speech SSL, and (2) the computational advantage and performance gain of PARP over baseline pruning methods. In particular, on the 10min Librispeech split without LM decoding, PARP discovers subnetworks from wav2vec 2. 0 with an absolute 10. 9%/12. 6% WER decrease compared to the full model. We further demonstrate the effectiveness of PARP via: cross-lingual pruning without any phone recognition degradation, the discovery of a multi-lingual subnetwork for 10 spoken languages in 1 finetuning run, and its applicability to pre-trained BERT/XLNet for natural language tasks1.

YNIMG Journal 2021 Journal Article

Speech frequency-following response in human auditory cortex is more than a simple tracking

• Ning Guo
• Xiaopeng Si
• Yang Zhang
• Yue Ding
• Wenjing Zhou
• Dan Zhang
• Bo Hong

The human auditory cortex is recently found to contribute to the frequency following response (FFR) and the cortical component has been shown to be more relevant to speech perception. However, it is not clear how cortical FFR may contribute to the processing of speech fundamental frequency (F0) and the dynamic pitch. Using intracranial EEG recordings, we observed a significant FFR at the fundamental frequency (F0) for both speech and speech-like harmonic complex stimuli in the human auditory cortex, even in the missing fundamental condition. Both the spectral amplitude and phase coherence of the cortical FFR showed a significant harmonic preference, and attenuated from the primary auditory cortex to the surrounding associative auditory cortex. The phase coherence of the speech FFR was found significantly higher than that of the harmonic complex stimuli, especially in the left hemisphere, showing a high timing fidelity of the cortical FFR in tracking dynamic F0 in speech. Spectrally, the frequency band of the cortical FFR was largely overlapped with the range of the human vocal pitch. Taken together, our study parsed the intrinsic properties of the cortical FFR and reveals a preference for speech-like sounds, supporting its potential role in processing speech intonation and lexical tones.

NeurIPS Conference 2021 Conference Paper

Understanding Interlocking Dynamics of Cooperative Rationalization

• Mo Yu
• Yang Zhang
• Shiyu Chang
• Tommi Jaakkola

Selective rationalization explains the prediction of complex neural networks by finding a small subset of the input that is sufficient to predict the neural model output. The selection mechanism is commonly integrated into the model itself by specifying a two-component cascaded system consisting of a rationale generator, which makes a binary selection of the input features (which is the rationale), and a predictor, which predicts the output based only on the selected features. The components are trained jointly to optimize prediction performance. In this paper, we reveal a major problem with such cooperative rationalization paradigm --- model interlocking. Inter-locking arises when the predictor overfits to the features selected by the generator thus reinforcing the generator's selection even if the selected rationales are sub-optimal. The fundamental cause of the interlocking problem is that the rationalization objective to be minimized is concave with respect to the generator’s selection policy. We propose a new rationalization framework, called A2R, which introduces a third component into the architecture, a predictor driven by soft attention as opposed to selection. The generator now realizes both soft and hard attention over the features and these are fed into the two different predictors. While the generator still seeks to support the original predictor performance, it also minimizes a gap between the two predictors. As we will show theoretically, since the attention-based predictor exhibits a better convexity property, A2R can overcome the concavity barrier. Our experiments on two synthetic benchmarks and two real datasets demonstrate that A2R can significantly alleviate the interlock problem and find explanations that better align with human judgments.

IJCAI Conference 2021 Conference Paper

User Retention: A Causal Approach with Triple Task Modeling

• Yang Zhang
• Dong Wang
• Qiang Li
• Yue Shen
• Ziqi Liu
• Xiaodong Zeng
• Zhiqiang Zhang
• Jinjie Gu

For many Internet companies, it has been an important focus to improve user retention rate. To achieve this goal, we need to recommend proper services in order to meet the demands of users. Unlike conventional click-through rate (CTR) estimation, there are lots of noise in the collected data when modeling retention, caused by two major issues: 1) implicit impression-revisit effect: users could revisit the APP even if they do not explicitly interact with the recommender system; 2) selection bias: recommender system suffers from selection bias caused by user's self-selection. To address the above challenges, we propose a novel method named UR-IPW (User Retention Modeling with Inverse Propensity Weighting), which 1) makes full use of both explicit and implicit interactions in the observed data. 2) models revisit rate estimation from a causal perspective accounting for the selection bias problem. The experiments on both offline and online environments from different scenarios demonstrate the superiority of UR-IPW over previous methods. To the best of our knowledge, this is the first work to model user retention by estimating the revisit rate from a causal perspective.

IROS Conference 2020 Conference Paper

Adaptive Informative Sampling with Environment Partitioning for Heterogeneous Multi-Robot Systems

• Yunfei Shi
• Ning Wang
• Jianmin Zheng
• Yang Zhang
• Sha Yi
• Wenhao Luo
• Katia P. Sycara

Multi-robot systems are widely used in environmental exploration and modeling, especially in hazardous environments. However, different types of robots are limited by different mobility, battery life, sensor type, etc. Heterogeneous robot systems are able to utilize various types of robots and provide solutions where robots are able to compensate each other with their different capabilities. In this paper, we consider the problem of sampling and modeling environmental characteristics with a heterogeneous team of robots. To utilize heterogeneity of the system while remaining computationally tractable, we propose an environmental partitioning approach that leverages various robot capabilities by forming a uniformly defined heterogeneity cost space. We combine with the mixture of Gaussian Processes model-learning framework to adaptively sample and model the environment in an efficient and scalable manner. We demonstrate our algorithm in field experiments with ground and aerial vehicles.

AAAI Conference 2020 Conference Paper

Crowd-Assisted Disaster Scene Assessment with Human-AI Interactive Attention

• Daniel (Yue) Zhang
• Yifeng Huang
• Yang Zhang
• Dong Wang

The recent advances of mobile sensing and artificial intelligence (AI) have brought new revolutions in disaster response applications. One example is disaster scene assessment (DSA) which leverages computer vision techniques to assess the level of damage severity of the disaster events from images provided by eyewitnesses on social media. The assessment results are critical in prioritizing the rescue operations of the response teams. While AI algorithms can significantly reduce the detection time and manual labeling cost in such applications, their performance often falls short of the desired accuracy. Our work is motivated by the emergence of crowdsourcing platforms (e. g. , Amazon Mechanic Turk, Waze) that provide unprecedented opportunities for acquiring human intelligence for AI applications. In this paper, we develop an interactive Disaster Scene Assessment (iDSA) scheme that allows AI algorithms to directly interact with humans to identify the salient regions of the disaster images in DSA applications. We also develop new incentive designs and active learning techniques to ensure reliable, timely, and costefficient responses from the crowdsourcing platforms. Our evaluation results on real-world case studies during Nepal and Ecuador earthquake events demonstrate that iDSA can significantly outperform state-of-the-art baselines in accurately assessing the damage of disaster scenes.

ICLR Conference 2020 Conference Paper

Deep Symbolic Superoptimization Without Human Knowledge

• Hui Shi
• Yang Zhang
• Xinyun Chen
• Yuandong Tian
• Jishen Zhao

Deep symbolic superoptimization refers to the task of applying deep learning methods to simplify symbolic expressions. Existing approaches either perform supervised training on human-constructed datasets that defines equivalent expression pairs, or apply reinforcement learning with human-defined equivalent trans-formation actions. In short, almost all existing methods rely on human knowledge to define equivalence, which suffers from large labeling cost and learning bias, because it is almost impossible to define and comprehensive equivalent set. We thus propose HISS, a reinforcement learning framework for symbolic super-optimization that keeps human outside the loop. HISS introduces a tree-LSTM encoder-decoder network with attention to ensure tractable learning. Our experiments show that HISS can discover more simplification rules than existing human-dependent methods, and can learn meaningful embeddings for symbolic expressions, which are indicative of equivalence.

EAAI Journal 2020 Journal Article

RETRACTED: A novel integrated price and load forecasting method in smart grid environment based on multi-level structure

• Yang Zhang
• Caibo Deng
• Ran Zhao
• Sebastian leto

This article has been retracted: please see Elsevier policy on article withdrawal (https: //www. elsevier. com/about/policies-and-standards/article-withdrawal). This article has been retracted at the request of the Editor-in-Chief. Post-publication, the editor discovered suspicious changes in authorship between the original submission and the revised version of this paper. In summary, the paper was submitted by a sole author, Sebastian Leto. During revision, the authors Yang Zhang (New First and Corresponding Author), Caibo Deng, and Ran Zhao, were added to the revised paper without explanation and without exceptional approval by the journal editor, which is contrary to the journal policy on changes to authorship. The editor reached out to the authors for an explanation, but they failed to provide a satisfactory explanation to these changes. In addition, it appears that Sebastian Leto was claiming an affiliation with Department of Elec. & Energy, Sunlife Co. , Baku, Azerbaijan. When questioned about providing convincing evidence of the existence and nature of this company, the author, Sebastian Leto failed to reply. Overall, the editor feels that the findings of the manuscript cannot be relied upon, and the article needs to be retracted.

NeurIPS Conference 2020 Conference Paper

The Lottery Ticket Hypothesis for Pre-trained BERT Networks

• Tianlong Chen
• Jonathan Frankle
• Shiyu Chang
• Sijia Liu
• Yang Zhang
• Zhangyang Wang
• Michael Carbin

In natural language processing (NLP), enormous pre-trained models like BERT have become the standard starting point for training on a range of downstream tasks, and similar trends are emerging in other areas of deep learning. In parallel, work on the lottery ticket hypothesis has shown that models for NLP and computer vision contain smaller matching subnetworks capable of training in isolation to full accuracy and transferring to other tasks. In this work, we combine these observations to assess whether such trainable, transferrable subnetworks exist in pre-trained BERT models. For a range of downstream tasks, we indeed find matching subnetworks at 40% to 90% sparsity. We find these subnetworks at (pre-trained) initialization, a deviation from prior NLP research where they emerge only after some amount of training. Subnetworks found on the masked language modeling task (the same task used to pre-train the model) transfer universally; those found on other tasks transfer in a limited fashion if at all. As large-scale pre-training becomes an increasingly central paradigm in deep learning, our results demonstrate that the main lottery ticket observations remain relevant in this context. Codes available at https: //github. com/VITA-Group/BERT-Tickets.

NeurIPS Conference 2019 Conference Paper

A Game Theoretic Approach to Class-wise Selective Rationalization

• Shiyu Chang
• Yang Zhang
• Mo Yu
• Tommi Jaakkola

Selection of input features such as relevant pieces of text has become a common technique of highlighting how complex neural predictors operate. The selection can be optimized post-hoc for trained models or incorporated directly into the method itself (self-explaining). However, an overall selection does not properly capture the multi-faceted nature of useful rationales such as pros and cons for decisions. To this end, we propose a new game theoretic approach to class-dependent rationalization, where the method is specifically trained to highlight evidence supporting alternative conclusions. Each class involves three players set up competitively to find evidence for factual and counterfactual scenarios. We show theoretically in a simplified scenario how the game drives the solution towards meaningful class-dependent rationales. We evaluate the method in single- and multi-aspect sentiment classification tasks and demonstrate that the proposed method is able to identify both factual (justifying the ground truth label) and counterfactual (countering the ground truth label) rationales consistent with human rationalization. The code for our method is publicly available.

IJCAI Conference 2019 Conference Paper

Fairwalk: Towards Fair Graph Embedding

• Tahleen Rahman
• Bartlomiej Surma
• Michael Backes
• Yang Zhang

Graph embeddings have gained huge popularity in the recent years as a powerful tool to analyze social networks. However, no prior works have studied potential bias issues inherent within graph embedding. In this paper, we make a first attempt in this direction. In particular, we concentrate on the fairness of node2vec, a popular graph embedding method. Our analyses on two real-world datasets demonstrate the existence of bias in node2vec when used for friendship recommendation. We, therefore, propose a fairness-aware embedding method, namely Fairwalk, which extends node2vec. Experimental results demonstrate that Fairwalk reduces bias under multiple fairness metrics while still preserving the utility.

ICML Conference 2018 Conference Paper

A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations

• Weili Nie
• Yang Zhang
• Ankit Patel

Backpropagation-based visualizations have been proposed to interpret convolutional neural networks (CNNs), however a theory is missing to justify their behaviors: Guided backpropagation (GBP) and deconvolutional network (DeconvNet) generate more human-interpretable but less class-sensitive visualizations than saliency map. Motivated by this, we develop a theoretical explanation revealing that GBP and DeconvNet are essentially doing (partial) image recovery which is unrelated to the network decisions. Specifically, our analysis shows that the backward ReLU introduced by GBP and DeconvNet, and the local connections in CNNs are the two main causes of compelling visualizations. Extensive experiments are provided that support the theoretical analysis.

IROS Conference 2018 Conference Paper

Fully Convolutional Grasp Detection Network with Oriented Anchor Box

• Xinwen Zhou
• Xuguang Lan
• Hanbo Zhang
• Zhiqiang Tian
• Yang Zhang
• Nanning Zheng 0001

In this paper, we present a real-time approach to predict multiple grasping poses for a parallel-plate robotic gripper using RGB images. A model with oriented anchor box mechanism is proposed and a new matching strategy is used during the training process. An end-to-end fully convolutional neural network is employed in our work. The network consists of two parts: the feature extractor and multi-grasp predictor. The feature extractor is a deep convolutional neural network. The multi-grasp predictor regresses grasp rectangles from predefined oriented rectangles, called oriented anchor boxes, and classifies the rectangles into graspable and ungraspable. On the standard Cornell Grasp Dataset, our model achieves an accuracy of 97. 74% and 96. 61% on image-wise split and object-wise split respectively, and outperforms the latest state-of-the-art approach by 1. 74% on image-wise split and 0. 51% on object-wise split.

NeurIPS Conference 2017 Conference Paper

Dilated Recurrent Neural Networks

• Shiyu Chang
• Yang Zhang
• Wei Han
• Mo Yu
• Xiaoxiao Guo
• Wei Tan
• Xiaodong Cui
• Michael Witbrock

Learning with recurrent neural networks (RNNs) on long sequences is a notoriously difficult task. There are three major challenges: 1) complex dependencies, 2) vanishing and exploding gradients, and 3) efficient parallelization. In this paper, we introduce a simple yet effective RNN connection structure, the DilatedRNN, which simultaneously tackles all of these challenges. The proposed architecture is characterized by multi-resolution dilated recurrent skip connections and can be combined flexibly with diverse RNN cells. Moreover, the DilatedRNN reduces the number of parameters needed and enhances training efficiency significantly, while matching state-of-the-art performance (even with standard RNN cells) in tasks involving very long-term dependencies. To provide a theory-based quantification of the architecture's advantages, we introduce a memory capacity measure, the mean recurrent length, which is more suitable for RNNs with long skip connections than existing measures. We rigorously prove the advantages of the DilatedRNN over other recurrent neural architectures. The code for our method is publicly available at https: //github. com/code-terminator/DilatedRNN.

YNICL Journal 2016 Journal Article

Effects of outcome on the covariance between risk level and brain activity in adolescents with internet gaming disorder

• Xin Qi
• Yongxin Yang
• Shouping Dai
• Peihong Gao
• Xin Du
• Yang Zhang
• Guijin Du
• Xiaodong Li

Individuals with internet gaming disorder (IGD) often have impaired risky decision-making abilities, and IGD-related functional changes have been observed during neuroimaging studies of decision-making tasks. However, it is still unclear how feedback (outcomes of decision-making) affects the subsequent risky decision-making in individuals with IGD. In this study, twenty-four adolescents with IGD and 24 healthy controls (HCs) were recruited and underwent functional magnetic resonance imaging while performing the balloon analog risk task (BART) to evaluate the effects of prior outcomes on brain activity during subsequent risky decision-making in adolescents with IGD. The covariance between risk level and activation of the bilateral ventral medial prefrontal cortex, left inferior frontal cortex, right ventral striatum (VS), left hippocampus/parahippocampus, right inferior occipital gyrus/fusiform gyrus and right inferior temporal gyrus demonstrated interaction effects of group by outcome (P <0. 05, AlphaSim correction). The regions with interactive effects were defined as ROI, and ROI-based intergroup comparisons showed that the covariance between risk level and brain activation was significantly greater in adolescents with IGD compared with HCs after a negative outcome occurred (P <0. 05). Our results indicated that negative outcomes affected the covariance between risk level and activation of the brain regions related to value estimation (prefrontal cortex), anticipation of rewards (VS), and emotional-related learning (hippocampus/parahippocampus), which may be one of the underlying neural mechanisms of disadvantageous risky decision-making in adolescents with IGD.

AAMAS Conference 2013 Conference Paper

Multimodal Intelligent Affect Detection with Kinect

• Yang Zhang
• Li Zhang
• Alamgir Hossain

Communication between human beings involves complex and rich means. In the past decades, computers have successfully supported human in a variety of tasks such as calculating and memorizing. However, when confronted with the demand of multimodal interaction with users, can these indispensable partners make us satisfied? This research might answer this question.

IJCAI Conference 2009 Conference Paper

• Yabin Zheng
• Zhiyuan Liu
• Maosong Sun
• Liyun Ru
• Yang Zhang

In this paper, we proposed a novel method to detect new words in domain-specific fields based on user behaviors. First, we select the most representative words from domain-specific lexicon. Then combining with user behaviors, we try to discover the potential experts in this field who use those terminologies frequently. Finally, we make further efforts to identify new words from behaviors of those experts. Words used much more frequently in this community than others are most probably new words. In brief, our method follows a collaborative filtering way: first from words to find professional experts, then from experts to discover new words, which is different from the traditional new word detection methods. Our method achieves up to 0. 86 in accuracy on a computer science related data set. Moreover, the proposed method can be easily extended to related words retrieval task. We compare our method with Google Sets and Bayesian Sets. Experiments show that our method and Bayesian Sets gives better results than Google Sets.

YNIMG Journal 2009 Journal Article

Neural signatures of phonetic learning in adulthood: A magnetoencephalography study

• Yang Zhang
• Patricia K. Kuhl
• Toshiaki Imada
• Paul Iverson
• John Pruitt
• Erica B. Stevens
• Masaki Kawakatsu
• Yoh'ichi Tohkura

The present study used magnetoencephalography (MEG) to examine perceptual learning of American English /r/ and /l/ categories by Japanese adults who had limited English exposure. A training software program was developed based on the principles of infant phonetic learning, featuring systematic acoustic exaggeration, multi-talker variability, visible articulation, and adaptive listening. The program was designed to help Japanese listeners utilize an acoustic dimension relevant for phonemic categorization of /r–l/ in English. Although training did not produce native-like phonetic boundary along the /r–l/ synthetic continuum in the second language learners, success was seen in highly significant identification improvement over twelve training sessions and transfer of learning to novel stimuli. Consistent with behavioral results, pre–post MEG measures showed not only enhanced neural sensitivity to the /r–l/ distinction in the left-hemisphere mismatch field (MMF) response but also bilateral decreases in equivalent current dipole (ECD) cluster and duration measures for stimulus coding in the inferior parietal region. The learning-induced increases in neural sensitivity and efficiency were also found in distributed source analysis using Minimum Current Estimates (MCE). Furthermore, the pre–post changes exhibited significant brain-behavior correlations between speech discrimination scores and MMF amplitudes as well as between the behavioral scores and ECD measures of neural efficiency. Together, the data provide corroborating evidence that substantial neural plasticity for second-language learning in adulthood can be induced with adaptive and enriched linguistic exposure. Like the MMF, the ECD cluster and duration measures are sensitive neural markers of phonetic learning.

YNIMG Journal 2005 Journal Article

Effects of language experience: Neural commitment to language-specific auditory patterns

• Yang Zhang
• Patricia K. Kuhl
• Toshiaki Imada
• Makoto Kotani
• Yoh'ichi Tohkura

Linguistic experience alters an individual's perception of speech. We here provide evidence of the effects of language experience at the neural level from two magnetoencephalography (MEG) studies that compare adult American and Japanese listeners' phonetic processing. The experimental stimuli were American English /ra/ and /la/ syllables, phonemic in English but not in Japanese. In Experiment 1, the control stimuli were /ba/ and /wa/ syllables, phonemic in both languages; in Experiment 2, they were non-speech replicas of /ra/ and /la/. The behavioral and neuromagnetic results showed that Japanese listeners were less sensitive to the phonemic /r–l/ difference than American listeners. Furthermore, processing non-native speech sounds recruited significantly greater brain resources in both hemispheres and required a significantly longer period of brain activation in two regions, the superior temporal area and the inferior parietal area. The control stimuli showed no significant differences except that the duration effect in the superior temporal cortex also applied to the non-speech replicas. We argue that early exposure to a particular language produces a “neural commitment” to the acoustic properties of that language and that this neural commitment interferes with foreign language processing, making it less efficient.