Arrow Research search

Author name cluster

Yang Zhou

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

69 papers
2 author rows

Possible papers

69

EAAI Journal 2026 Journal Article

A high-performance memetic algorithm for integrated process planning and shop floor scheduling considering heterogeneous automated guided vehicle transportation system

  • Yang Zhou
  • Guiliang Gong
  • Xiahui Liu
  • Zhipeng Yuan
  • Hongbo Huang
  • Ao He
  • Jie Li

Integrating transportation decisions into Integrated Process Planning and Shop Scheduling (IPPS) enables tighter coordination between machining operations and material handling, leading to more efficient production schedules. However, most existing IPPS studies assume homogeneous or unlimited transportation resources, and thus overlook the practical scheduling complexity introduced by heterogeneous and capacity-limited automated guided vehicle (AGV) fleets. This study proposes an IPPS model with a heterogeneous AGV transportation system (IPPSHT), in which process route selection, operation sequencing, machine assignment, and AGV allocation are jointly optimized under functional compatibility constraints between AGVs and machines. To solve the resulting bi-objective problem of minimizing makespan and total energy consumption across machining and transportation stages, we develop a high-performance memetic algorithm that combines tailored encoding/decoding, dominance-based selection, and problem-aware local improvement. Extensive experiments on 60 benchmark instances and an engineering case study demonstrate that the proposed method consistently outperforms three representative multi-objective evolutionary algorithms in terms of solution quality and statistical significance, indicating its effectiveness for complex manufacturing settings with heterogeneous material-handling resources.

Details DOI

JBHI Journal 2026 Journal Article

A Real-Time Scale-Robust Network for Glottis Segmentation in Nasal Transnasal Intubation

  • Yang Zhou
  • Chaoyong Zhang
  • Ruoyi Hao
  • Huilin Pan
  • Yang Zhang
  • Hongliang Ren

Nasotracheal intubation (NTI) is a critical clinical procedure for establishing and maintaining patient airway patency. Machine-assisted NTI has emerged as a pivotal approach for optimizing procedural efficiency and minimizing manual intervention. However, visual detection algorithms employed for NTI navigation encounter significant challenges, including complex anatomical environments and suboptimal illumination conditions surrounding the glottis. Additionally, the glottis presents considerable scale variability throughout the procedure, initially appearing as a small, difficult-to-capture structure before expanding to occupy nearly the entire field of view. Moreover, traditional visual detection methods often have high computational costs, making real-time, high-precision detection on portable devices challenging. To enhance NTI efficacy and address these challenges, this paper proposes a novel glottis segmentation framework optimized for vision-assisted NTI applications. First, we designed a lightweight, multi-receptive field feature extraction module to reduce intra-class differences, achieving robustness to scale variations of the glottis. This module was then stacked to form the backbone and neck of our network. Subsequently, we developed an advanced label assignment method and redefined the number of samples to further reduce intra-class differences and enhance accuracy in the complex NTI environment. Experiments on three distinct datasets demonstrate that our network surpasses state-of-the-art algorithms, achieving a segmentation mDice of 92. 9% with a compact model size of 19 MB and an inference speed exceeding 170 frames per second.

Details DOI

AAAI Conference 2026 Conference Paper

CyPortQA: Benchmarking Multimodal Large Language Models for Cyclone Preparedness in Port Operation

  • Chenchen Kuai
  • Chenhao Wu
  • Yang Zhou
  • Bruce Wang
  • Tianbao Yang
  • Zhengzhong Tu
  • Zihao Li
  • Yunlong Zhang

As tropical cyclones intensify and track forecasts become increasingly uncertain, U.S. ports face heightened supply-chain risk under extreme weather conditions. Port operators need to rapidly synthesize diverse multimodal forecast products, such as probabilistic wind maps, track cones, and official advisories, into clear, actionable guidance as cyclones approach. Multimodal large language models (MLLMs) offer a powerful means to integrate these heterogeneous data sources alongside broader contextual knowledge, yet their accuracy and reliability in the specific context of port cyclone preparedness have not been rigorously evaluated. To fill this gap, we introduce CyPortQA, the first multimodal benchmark tailored to port operations under cyclone threat. CyPortQA assembles 2,917 real-world disruption scenarios from 2015 through 2023, spanning 145 U.S. principal ports and 90 named storms. Each scenario fuses multi-source data (i.e., tropical cyclone products, port operational impact records, and port condition bulletins) and is expanded through an automated pipeline into 117,178 structured question–answer pairs. Using this benchmark, we conduct extensive experiments on diverse MLLMs, including both open-source and proprietary model. MLLMs demonstrate great potential in situation understanding but still face considerable challenges in reasoning tasks, including potential impact estimation and decision reasoning.

PDF Details DOI

AAAI Conference 2026 Conference Paper

FracSegmentator: Fracture Instance Segmentation with Trauma-Prior-Guided Contrastive Learning

  • Yanzhen Liu
  • Sutuke Yibulayimu
  • Yang Zhou
  • Yudi Sang
  • Yu Wang

Fracture injuries often lead to complex bone fragmentations, posing significant challenges for accurate segmentation in surgical planning and trauma assessment. Manual annotation of each fragment is time-consuming and inconsistent, while existing automated methods often fail to separate individual fragments due to the wide variation in fracture types, irregular fracture surface, and close inter-fragment contact. To address these challenges, we introduce FracSegmentator, a deep learning approach for bone fragment instance segmentation. The model takes extracted bone regions in CT as input and isolates individual fragments by identifying fracture surfaces and separating closely contacting structures. Central to our approach is a Trauma-Prior-Guided Contrastive Learning module, which incorporates clinical knowledge through memory-based attention to better distinguish fractured surfaces from healthy regions. We evaluate FracSegmentator on four datasets that cover a range of anatomical sites and fracture patterns. The method achieves state-of-the-art results across all datasets and demonstrates strong generalization capabilities. By delivering accurate and efficient fragment-level segmentation, FracSegmentator supports critical downstream tasks such as automated fracture diagnosis, surgical planning, and preoperative reduction simulation.

PDF Details DOI

AAAI Conference 2026 Conference Paper

FusedRec: Fused Embedding Communication for Distributed Recommendation Training on GPUs

  • Xuanteng Huang
  • Fan Li
  • Riyang Hu
  • Jianchang Zhang
  • Yuan Peng
  • Yang Zhou
  • Fangying Chen
  • Xianwei Zhang

Recent years have witnessed the wide adoption of deep learning recommendation models (DLRMs) for many online services. Unlike traditional DNN training, DLRMs leverage massive embeddings to represent sparse features, which are stored in distributed GPUs following the model parallel paradigm. Existing approaches adopt deduplication to eliminate replicated embeddings involved in AltoAll transfers to avoid unnecessary communication. In our practices, we have observed that such a deduplication design exacerbates interconnect inefficiency due to the fragmented embedding transfers with reduced message sizes, hindering the performance of distributed DLRM training. This paper introduces FusedRec, a fused embedding communication and lookup mechanism to tackle the inefficiency due to deduplication. By seeking the opportunities to fuse embeddings from multiple categories into a group, FusedRec conducts the communication in a combined shot to alleviate bandwidth under-utilization. Meanwhile, a categorical-aware hashing algorithm is integrated into FusedRec to retain the category information during lookup without extra communication. Combining with efficient unique and recovery operations, comprehensive results show FusedRec achieves a 37.8% throughput speedup in average compared to the SOTA industry implementation, without hurting the recommendation qualities of our in-house models used in online production environments.

PDF Details DOI

TCS Journal 2026 Journal Article

k-Submodular and approximately non-k-submodular maximization under p-system and ℓ knapsack constraints

  • Hanlu Ye
  • Heqing Li
  • Min Li
  • Yang Zhou
  • Qian Liu

This paper addresses the problem of k-submodular and approximately non-k-submodular maximization under p-system and ℓ knapsack constraints. For monotone k-submodular functions, we first propose a greedy algorithm, achieving a 1 ( 1 + ϵ ′ ) ( 1 + p + 2 ℓ ) -approximation and a 1 ( 1 + ϵ ′ ) ( 2 + p + 2 ℓ ) -approximation for non-monotone case, with the O ( n 2 ( 1 + k ) log ( 2 n ) log ( 1 + ϵ ′ ) ) time complexity, where ϵ′ is a very small positive number. We further introduce an improved algorithm that enhances the approximation ratio to 1 ( 1 + ϵ ′ + ϵ ′ 2 ) ( 1 + p + 7 4 ℓ ) and 1 ( 1 + ϵ ′ + ϵ ′ 2 ) ( 2 + p + 7 4 ℓ ), respectively, while reducing the time complexity to O ( n k log n ϵ ′ log ( 2 n ) ). For monotone k-submodular functions with curvature c, we obtain an approximation result of 1 ( 1 + ϵ ′ ) ( p + c + ϵ ′ + 7 4 ℓ ). Additionally, we provide an approximation guarantee of min { 1, 1 α ( 1 + ϵ ′ ) } 1 + 1 + ϵ α 2 ( 1 − ϵ ) [ ( 1 + ϵ ′ ) ( p + α ϵ ′ ) + 7 4 ℓ ] for ϵ-approximately α-weakly diminishing returns functions.

Details DOI

AAAI Conference 2026 Conference Paper

Note2Chat: Improving LLMs for Multi-Turn Clinical History Taking Using Medical Notes

  • Yang Zhou
  • Zhenting Sheng
  • Mingrui Tan
  • Yuting Song
  • Jun Zhou
  • Yu Heng Kwan
  • Lian Leng Low
  • Yang Bai

Effective clinical history taking is a foundational yet underexplored component of clinical reasoning. While large language models (LLMs) have shown promise on static benchmarks, they often fall short in dynamic, multi-turn diagnostic settings that require iterative questioning and hypothesis refinement. To address this gap, we propose Note2Chat, a note-driven framework that trains LLMs to conduct structured history taking and diagnosis by learning from widely available medical notes. Instead of relying on scarce and sensitive dialogue data, we convert real-world medical notes into high-quality doctor-patient dialogues using a decision tree-guided generation and refinement pipeline. We then propose a three-stage fine-tuning strategy combining supervised learning, simulated data augmentation, and preference learning. Furthermore, we propose a novel single-turn reasoning paradigm that reframes history taking as a sequence of single-turn reasoning problems. This design enhances interpretability and enables local supervision, dynamic adaptation, and greater sample efficiency. Experimental results show that our method substantially improves clinical reasoning, achieving gains of +16.9 F1 and +21.0 Top-1 diagnostic accuracy over GPT-4o.

PDF Details DOI

AAAI Conference 2026 Conference Paper

UVLM: Benchmarking Video Language Model for Underwater World Understanding

  • Xizhe Xue
  • Yang Zhou
  • Dawei Yan
  • Lijie Tao
  • Junjie Li
  • Ying Li
  • Haokui Zhang
  • Rong Xiao

Recently, video-language models (VidLMs) have gained widespread attention and adoption. However, existing works primarily focus on terrestrial scenarios, overlooking the highly demanding application needs of underwater observation. To overcome this gap, we introduce UVLM, an under water observation benchmark which is build through a collaborative approach combining human expertise and AI models. To ensure data quality, we have conducted in-depth considerations from multiple perspectives. First, to address the unique challenges of underwater environments, we selected videos that represent typical underwater challenges including light variations, water turbidity, and diverse viewing angles to construct the dataset. Second, to ensure data diversity, the dataset covers a wide range of frame rates, resolutions, 419 classes of marine animals, and various static plants and terrains. Next, for task diversity, we adopted a structured design where observation targets are categorized into two major classes: biological and environmental. Each category includes content observation and change/action observation, totaling 20 subtask types. Finally, we designed several challenging evaluation metrics to enable quantitative comparison and analysis of different methods. Experiments on two representative VidLMs demonstrate that fine-tuning VidLMs on UVLM significantly improves underwater world understanding while also showing potential for slight improvements on existing in-air VidLM benchmarks.

PDF Details DOI

ICLR Conference 2025 Conference Paper

A Robust Method to Discover Causal or Anticausal Relation

  • Yu Yao 0005
  • Yang Zhou
  • Bo Han 0003
  • Mingming Gong
  • Kun Zhang 0001
  • Tongliang Liu

Understanding whether the data generative process follows causal or anticausal relations is important for many applications. Existing causal discovery methods struggle with high-dimensional perceptual data such as images. Moreover, they require well-labeled data, which may not be feasible due to measurement error. In this paper, we propose a robust method to detect whether the data generative process is causal or anticausal. To determine the causal or anticausal relation, we identify an asymmetric property: under the causal relation, the instance distribution does not contain information about the noisy class-posterior distribution. We also propose a practical method to verify this via a noise injection approach. Our method is robust to label errors and is designed to handle both large-scale and high-dimensional datasets effectively. Both theoretical analyses and empirical results on a variety of datasets demonstrate the effectiveness of our proposed method in determining the causal or anticausal direction of the data generative process.

Details

NeurIPS Conference 2025 Conference Paper

Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts

  • Haizhong Zheng
  • Yang Zhou
  • Brian Bartoldson
  • Bhavya Kailkhura
  • Fan Lai
  • Jiawei Zhao
  • Beidi Chen

Reinforcement learning, such as PPO and GRPO, has powered recent breakthroughs in LLM reasoning. Scaling rollout to sample more prompts enables models to selectively use higher-quality data for training, which can stabilize RL training and improve model performance, but at the cost of significant computational overhead. In this paper, we first show that a substantial portion of this overhead can be avoided by skipping uninformative prompts before rollout. Our analysis of reward dynamics reveals a strong temporal consistency in prompt value: prompts that are uninformative in one epoch of training are likely to remain uninformative in near future epochs. Based on these insights, we propose GRESO (GRPO with Efficient Selective Rollout), an online, lightweight pre-rollout filtering algorithm that predicts and skips uninformative prompts using reward training dynamics. By evaluating GRESO on a broad range of math reasoning benchmarks and models, like Qwen2. 5-Math-1. 5B, DeepSeek-R1-Distill-Qwen-1. 5B, Qwen2. 5-Math-7B, Qwen2. 5-14B, and Qwen2. 5-32B, we show that GRESO achieves up to 2. 4x wall-clock time speedup in rollout and up to 2. 0x speedup in total training time without accuracy degradation. We make our code publicly available at https: //github. com/Infini-AI-Lab/GRESO/.

PDF Details

TMLR Journal 2025 Journal Article

AutoTrust: Benchmarking Trustworthiness in Large Vision Language Models for Autonomous Driving

  • Shuo Xing
  • Hongyuan Hua
  • Xiangbo Gao
  • Shenzhe Zhu
  • Renjie Li
  • Kexin Tian
  • Xiaopeng Li
  • Heng Huang

Recent advancements in large vision language models (VLMs) tailored for autonomous driving (AD) have shown strong scene understanding and reasoning capabilities, making them undeniable candidates for end-to-end driving systems. However, limited work exists on studying the trustworthiness of DriveVLMs—a critical factor that directly impacts public transportation safety. In this paper, we introduce AutoTrust, a comprehensive trustworthiness benchmark for large vision-language models in autonomous driving (DriveVLMs), considering diverse perspectives---including trustfulness, safety, robustness, privacy, and fairness. We constructed the largest visual question-answering dataset for investigating trustworthiness issues in driving scenarios, comprising over 10k unique scenes and 18k queries. We evaluated six publicly available VLMs, spanning from generalist to specialist, from open-source to commercial models. Our exhaustive evaluations have unveiled previously undiscovered vulnerabilities of DriveVLMs to trustworthiness threats. Specifically, we found that the general VLMs like LLaVA-v1.6 and GPT-4o-mini surprisingly outperform specialized models fine-tuned for driving in terms of overall trustworthiness. DriveVLMs like DriveLM-Agent are particularly vulnerable to disclosing sensitive information. Additionally, both generalist and specialist VLMs remain susceptible to adversarial attacks and struggle to ensure unbiased decision-making across diverse environments and populations. Our findings call for immediate and decisive action to address the trustworthiness of DriveVLMs--an issue of critical importance to public safety and the welfare of all citizens relying on autonomous transportation systems. We release all the codes and datasets in https://github.com/taco-group/AutoTrust.

PDF Details

NeurIPS Conference 2025 Conference Paper

Bridging Symmetry and Robustness: On the Role of Equivariance in Enhancing Adversarial Robustness

  • Longwei Wang
  • Ifrat Ikhtear Uddin
  • Prof. KC Santosh (PhD)
  • Chaowei Zhang
  • Xiao Qin
  • Yang Zhou

Adversarial examples reveal critical vulnerabilities in deep neural networks by exploiting their sensitivity to imperceptible input perturbations. While adversarial training remains the predominant defense strategy, it often incurs significant computational cost and may compromise clean-data accuracy. In this work, we investigate an architectural approach to adversarial robustness by embedding group-equivariant convolutions—specifically, rotation- and scale-equivariant layers—into standard convolutional neural networks (CNNs). These layers encode symmetry priors that align model behavior with structured transformations in the input space, promoting smoother decision boundaries and greater resilience to adversarial attacks. We propose and evaluate two symmetry-aware architectures: a parallel design that processes standard and equivariant features independently before fusion, and a cascaded design that applies equivariant operations sequentially. Theoretically, we demonstrate that such models reduce hypothesis space complexity, regularize gradients, and yield tighter certified robustness bounds under the CLEVER (Cross Lipschitz Extreme Value for nEtwork Robustness) framework. Empirically, our models consistently improve adversarial robustness and generalization across CIFAR-10, CIFAR-100, and CIFAR-10C under both FGSM and PGD attacks, without requiring adversarial training. These findings underscore the potential of symmetry-enforcing architectures as efficient and principled alternatives to data augmentation-based defenses.

PDF Details

IROS Conference 2025 Conference Paper

Causal-Planner: Causal Interaction Disentangling with Episodic Memory Gating for Autonomous Planning

  • Yibo Yuan
  • Jianwu Fang
  • Yang Zhou
  • Zhao Yang
  • Chen Lv 0001
  • Jianru Xue

Autonomous vehicle trajectory planning faces significant challenges in dynamic traffic environments due to the complex and mixed causal relationships between critical scene elements (e. g. , pedestrians, vehicles, road markings) and safe decision-making. To identify the causal factors influencing planning outcomes, we propose Causal-Planner, which disentangles the scene interaction graph into causal and confounding components via attention-based adversarial graph learning. Additionally, we introduce a long-short-term episodic memory gating (LSTEM) module that enhances causal interaction disentangling by adaptively capturing evolving causal relationships in dynamic scenarios through bidirectional gated memory fusion. Extensive experiments on the nuPlan dataset suggest that Causal-Planner achieves competitive performance, performing well in both Test-random and Test-hard scenarios under open-loop and closed-loop evaluations. The code will be publicly available at https://github.com/Yyb-XJTU/Causal-Planner.

Details

IROS Conference 2025 Conference Paper

CLAP: A Closed-Loop Diffusion Transformer Action Foundation Model for Robotic Manipulation

  • Mu Li
  • Yubo Dong
  • Yang Zhou
  • Chenguang Yang 0001

The development of large Vision-Language-Action (VLA) models has enhanced the robot’s ability to manipulate objects in unseen scenarios based on language instructions. While existing VLAs have demonstrated promise in various scenarios, they still struggle with effective multi-modal data feature extraction and lack a closed-loop inference framework. In this paper, we propose an advanced VLA model. Unlike previous works that repurpose VLM for action prediction using simple action quantization, we componentized the VLA architecture with a specialized action module conditioned on the model output and a critic module for inference. We demonstrate the performance improvement of diffusion action transformers in modeling continuous temporal actions, with the critic module applied during inference to form a closed-loop model. Extensive experiments on real robots demonstrate that our model significantly outperforms existing methods, with the ability to handle complex, high-precision tasks and generalize to unseen objects and backgrounds.

Details

IJCAI Conference 2025 Conference Paper

Faster Annotation for Elevation-Guided Flood Extent Mapping by Consistency-Enhanced Active Learning

  • Saugat Adhikari
  • Da Yan
  • Tianyang Wang
  • Landon Dyken
  • Sidharth Kumar
  • Lyuheng Yuan
  • Akhlaque Ahmad
  • Jiao Han

Flood extent mapping is crucial for disaster response and damage assessment. While Earth imagery and terrain data (in the form of DEM) are now readily available, there are few flood annotation data for training machine learning models, which hinders the automated mapping of flooded areas. We propose ALFA, an interactive active-learning-based approach to minimize the annotators' efforts when preparing the ground-truth flood map in a satellite image. ALFA calibrates the prediction consistency of a segmentation model (1) across training cycles and (2) for various data augmentations. The two consistencies are integrated into the design of both the acquisition function and the loss function to enhance the robustness of active learning with limited annotation inputs. ALFA recommends those superpixels that the underlying model is most uncertain about, and users can annotate their pixels with minimal clicks with the help of elevation guidance. Extensive experiments on various regions hit by flooding show that we can improve the annotation time from hours to around 20 minutes. ALFA is open sourced at https: //github. com/saugatadhikari/alfa.

PDF Details DOI

ICML Conference 2025 Conference Paper

GSM-∞: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?

  • Yang Zhou
  • Hongyi Liu
  • Zhuoming Chen
  • Yuandong Tian
  • Beidi Chen

Recently, long-context large language models (LLMs) have shown strong performance in information retrieval and long-document QA. However, to tackle the most challenging intellectual problems, LLMs must reason effectively in long and complex contexts (e. g. , frontier mathematical research). Studying how LLMs handle increasing reasoning complexity and context length is essential, yet existing benchmarks lack a solid basis for quantitative evaluation. Inspired by the abstraction of GSM-8K problems as computational graphs—and the ability to introduce noise by adding unnecessary nodes and edges—we develop a grade-school math problem generator capable of producing arithmetic problems with infinite difficulty and context length under fine-grained control. Using our newly synthesized GSM-$\infty$ benchmark, we comprehensively evaluate existing LLMs. We find a consistent sigmoid decline in reasoning performance as complexity increases, along with a systematic inference scaling trend: exponentially increasing inference computation yields only linear performance gains. These findings underscore the fundamental limitations of current long-context LLMs and the key challenges in scaling reasoning capabilities. Our GSM-$\infty$ benchmark provides a scalable and controllable testbed for systematically studying and advancing LLM reasoning in long and complex contexts.

Details

ICLR Conference 2025 Conference Paper

HiSplat: Hierarchical 3D Gaussian Splatting for Generalizable Sparse-View Reconstruction

  • Shengji Tang
  • Weicai Ye
  • Peng Ye 0006
  • Weihao Lin 0002
  • Yang Zhou
  • Tao Chen 0003
  • Wanli Ouyang

Reconstructing 3D scenes from multiple viewpoints is a fundamental task in stereo vision. Recently, advances in generalizable 3D Gaussian Splatting have enabled high-quality novel view synthesis for unseen scenes from sparse input views by feed-forward predicting per-pixel Gaussian parameters without extra optimization. However, existing methods typically generate single-scale 3D Gaussians, which lack representation of both large-scale structure and texture details, resulting in mislocation and artefacts. In this paper, we propose a novel framework, HiSplat, which introduces a hierarchical manner in generalizable 3D Gaussian Splatting to construct hierarchical 3D Gaussians via a coarse-to-fine strategy. Specifically, HiSplat generates large coarse-grained Gaussians to capture large-scale structures, followed by fine-grained Gaussians to enhance delicate texture details. To promote inter-scale interactions, we propose an Error Aware Module for Gaussian compensation and a Modulating Fusion Module for Gaussian repair. Our method achieves joint optimization of hierarchical representations, allowing for novel view synthesis using only two-view reference images. Comprehensive experiments on various datasets demonstrate that HiSplat significantly enhances reconstruction quality and cross-dataset generalization compared to prior single-scale methods. The corresponding ablation study and analysis of different-scale 3D Gaussians reveal the mechanism behind the effectiveness. Code is at https://github.com/Open3DVLab/HiSplat.

Details

NeurIPS Conference 2025 Conference Paper

Jury-and-Judge Chain-of-Thought for Uncovering Toxic Data in 3D Visual Grounding

  • Kaixiang Huang
  • Qifeng Zhang
  • Jin Wang
  • Jingru Yang
  • Yang Zhou
  • Huan Yu
  • Guodong Lu
  • Shengfeng He

3D Visual Grounding (3DVG) faces persistent challenges due to coarse scene-level observations and logically inconsistent annotations, which introduce ambiguities that compromise data quality and hinder effective model supervision. To address these challenges, we introduce Refer-Judge, a novel framework that harnesses the reasoning capabilities of Multimodal Large Language Models (MLLMs) to identify and mitigate toxic data. At the core of Refer-Judge is a Jury-and-Judge Chain-of-Thought paradigm, inspired by the deliberative process of the judicial system. This framework targets the root causes of annotation noise: jurors collaboratively assess 3DVG samples from diverse perspectives, providing structured, multi-faceted evaluations. Judges then consolidate these insights using a Corroborative Refinement strategy, which adaptively reorganizes information to correct ambiguities arising from biased or incomplete observations. Through this two-stage deliberation, Refer-Judge significantly enhances the reliability of data judgments. Extensive experiments demonstrate that our framework not only achieves human-level discrimination at the scene level but also improves the performance of baseline algorithms via data purification. Code is available at https: //github. com/Hermione-HKX/Refer_Judge.

PDF Details

ICLR Conference 2025 Conference Paper

MagicPIG: LSH Sampling for Efficient LLM Generation

  • Zhuoming Chen
  • Ranajoy Sadhukhan
  • Zihao Ye 0001
  • Yang Zhou
  • Jianyu Zhang
  • Niklas Nolte
  • Yuandong Tian
  • Matthijs Douze

Large language models (LLMs) with long context windows have gained significant attention. However, the KV cache, stored to avoid re-computation, becomes a bottleneck. Various dynamic sparse or TopK-based attention approximation methods have been proposed to leverage the common insight that attention is sparse. In this paper, we first show that TopK attention itself suffers from quality degradation in certain downstream tasks because attention is not always as sparse as expected. Rather than selecting the keys and values with the highest attention scores, sampling with theoretical guarantees can provide a better estimation for attention output. To make the sampling-based approximation practical in LLM generation, we propose MagicPIG, a heterogeneous system based on Locality Sensitive Hashing (LSH). MagicPIG significantly reduces the workload of attention computation while preserving high accuracy for diverse tasks. MagicPIG stores the LSH hash tables and runs the attention computation on the CPU, which allows it to serve longer contexts and larger batch sizes with high approximation accuracy. MagicPIG can improve decoding throughput by up to $5\times$ across various GPU hardware and achieve 54ms decoding latency on a single RTX 4090 for Llama-3.1-8B-Instruct model with a context of 96k tokens.

Details

NeurIPS Conference 2025 Conference Paper

PAC-Bayes Bounds for Multivariate Linear Regression and Linear Autoencoders

  • Ruixin Guo
  • Ruoming Jin
  • Xinyu Li
  • Yang Zhou

Linear Autoencoders (LAEs) have shown strong performance in state-of-the-art recommender systems. However, this success remains largely empirical, with limited theoretical understanding. In this paper, we investigate the generalizability -- a theoretical measure of model performance in statistical learning -- of multivariate linear regression and LAEs. We first propose a PAC-Bayes bound for multivariate linear regression, extending the earlier bound for single-output linear regression by Shalaeva et al. , and establish sufficient conditions for its convergence. We then show that LAEs, when evaluated under a relaxed mean squared error, can be interpreted as constrained multivariate linear regression models on bounded data, to which our bound adapts. Furthermore, we develop theoretical methods to improve the computational efficiency of optimizing the LAE bound, enabling its practical evaluation on large models and real-world datasets. Experimental results demonstrate that our bound is tight and correlates well with practical ranking metrics such as Recall@K and NDCG@K.

PDF Details

EAAI Journal 2025 Journal Article

Plug-and-play dynamic optimization for three-dimensional Gaussian generation

  • Qixuan Li
  • Haoyang Li
  • Chao Wang
  • Yang Zhou
  • Yan Peng

Recent advancements in Three-Dimensional (3D) asset generation have demonstrated remarkable progress in generation efficiency, enabling transformative applications across creative industries and mission-critical domains including autonomous systems. Current 3D asset generation primarily employs Score Distillation Sampling (SDS) to derive 3D priors from Two-Dimensional (2D) diffusion models. While this contribution ensures high generation quality, it is time-consuming. Recent methods have utilized 3D Gaussian Splatting for image rendering, which, despite enhancing generation speed, compromised on quality. Our method aims to balance the quality and speed of 3D asset generation by designing a plug-and-play optimization process that combines the strengths of both methods. We propose a rapid 3D Gaussian generation framework that begins with constructing a pipeline to generate multi-view images from text input using pre-trained generative models. Then our method utilizes 3D Gaussian Splatting for quick 3D asset initialization and subsequently performs detail optimization using Gaussian Filter and SDS-based 2D diffusion model optimizer. Additionally, we have optimized the loss function for 3D Gaussian Splatting and ensured the entire optimization process is plug-and-play, offering high generation quality and speed. Our method demonstrates strong adaptability in representative single-object 3D Gaussian generation tasks, indicating promising generalization potential. Achieving high-quality 3D generation on a single Graphics Processing Unit (GPU), our framework outperforms most popular optimization-based models in generation speed (5 × speedup +). Furthermore, when juxtaposed with the latest inference-based models, our optimization architecture offers a notable enhancement in generation quality (Contrastive Language-Image Pre-Training Score 33. 8 vs. 27. 3) within an acceptable amount of time.

Details DOI

AAAI Conference 2025 Conference Paper

Privacy-Preserving V2X Collaborative Perception Integrating Unknown Collaborators

  • Bin Lu
  • Xinyu Xiao
  • Changzhou Zhang
  • Yang Zhou
  • Zhiyu Xiang
  • Hangguan Shan
  • Eryun Liu

Vehicle-to-everything (V2X) collaborative perception has recently gained increasing attention in autonomous driving due to its ability to enhance scene understanding by integrating information from other collaborators, e.g. vehicles or infrastructure. Existing algorithms usually share deep features to achieve a trade-off between accuracy and bandwidth. However, most of these methods require joint training of all agents, which results in privacy leakage and is impractical and unacceptable in the real world. Sharing prediction results seems to be a direct solution, but its performance is suboptimal and sensitive to localization noise and communication delay. In this paper, we propose a privacy-preserving collaborative perception framework, where each agent is separately trained with its own dataset and the ego vehicle needs to integrate with completely unknown collaborators. Specifically, we propose MSD, a multi-scale feature fusion method combined with deformable attention, to better fuse features of different agents. We also propose a plug-in domain adapter to align the features from unknown collaborators to ego-domain. Extensive experiments on the challenging DAIR-V2X and V2V4Real demonstrate that: 1) MSD achieves remarkable performance, outperforming others by at least 2.8% and 6.7% in AP0.7 on DAIR-V2X and V2V4Real, respectively; 2) After domain adaptation, it significantly outperforms the No Fusion, Late Fusion scenarios and can approach or even surpass the performance of joint training. We truly achieves privacy-preserving collaboration, providing a new paradigm for the study of collaborative perception, which is crucial for practical applications.

PDF Details DOI

AAAI Conference 2025 System Paper

Rewind and Render: Towards Factually Accurate Text-to-Video Generation with Distilled Knowledge Retrieval

  • Daniel Lee
  • Arjun Chandra
  • Yang Zhou
  • Yunyao Li
  • Simone Conia

Text-to-Video (T2V) models, despite recent advancements, struggle with factual accuracy, especially for knowledge-dense content. We introduce FACT-V (Factual Accuracy in Content Translation to Video), a system integrating multi-source knowledge retrieval into T2V pipelines. FACT-V offers two key benefits: i) improved factual accuracy of generated videos through dynamically retrieved information, and ii) increased interpretability by providing users with the augmented prompt information. A preliminary evaluation demonstrates the potential of knowledge-augmented approaches in improving the accuracy and reliability of T2V systems, particularly for entity-specific or time-sensitive prompts.

PDF Details DOI

AAAI Conference 2025 Conference Paper

Self-Evolutionary Large Language Models Through Uncertainty-Enhanced Preference Optimization

  • Jianing Wang
  • Yang Zhou
  • Xiaocheng Zhang
  • Mengjiao Bao
  • Peng Yan

Iterative preference optimization has recently become one of the de-facto training paradigms for large language models (LLMs), but the performance is still underwhelming due to too much noisy preference data yielded in the loop. To combat this issue, we present an Uncertainty-enhanced Preference Optimization (UPO) framework to make the LLM self-evolve with reliable feedback. The key idea is mitigating the noisy preference pairs derived from the current policy and reward models by performing pair-wise uncertainty estimation and judiciously reliable feedback sampling. To reach this goal, we thus introduce an estimator model, which incorporates Monte Carlo (MC) dropout in Bayesian neural network (BNN) to perform uncertainty estimation for the batch of preference pairs. Compared to the existing methods that directly filter generated responses based on the reward score, the estimator focuses on the model uncertainty in a pair-wise manner and effectively bypasses the confirmation bias problem of the reward model. Additionally, we also propose an uncertainty-enhanced self-evolution algorithm to better improve the LLM robustly align with these reliable feedback data. Extensive experiments over multiple benchmarks demonstrate our framework substantially improves the performance of iterative preference optimization.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

SeRL: Self-play Reinforcement Learning for Large Language Models with Limited Data

  • Wenkai Fang
  • Shunyu Liu
  • Yang Zhou
  • Kongcheng Zhang
  • Tongya Zheng
  • Kaixuan Chen
  • Mingli Song
  • Dacheng Tao

Recent advances have demonstrated the effectiveness of Reinforcement Learning (RL) in improving the reasoning capabilities of Large Language Models (LLMs). However, existing works inevitably rely on high-quality instructions and verifiable rewards for effective training, both of which are often difficult to obtain in specialized domains. In this paper, we propose Self-play Reinforcement Learning (SeRL) to bootstrap LLM training with limited initial data. Specifically, SeRL comprises two complementary modules: self-instruction and self-rewarding. The former module generates additional instructions based on the available data at each training step, employing comprehensive online filtering strategies to ensure instruction quality, diversity, and difficulty. The latter module introduces a simple yet effective majority-voting mechanism to estimate response rewards for additional instructions, eliminating the need for external annotations. Finally, SeRL performs conventional RL based on the generated data, facilitating iterative self-play learning. Extensive experiments on various reasoning benchmarks and across different LLM backbones demonstrate that the proposed SeRL yields results superior to its counterparts and achieves performance on par with those obtained by high-quality data with verifiable rewards. Our code is available at https: //github. com/wantbook-book/SeRL.

PDF Details

AAAI Conference 2025 Conference Paper

ShotVL: Human-Centric Highlight Frame Retrieval via Language Queries

  • Wangyu Xue
  • Chen Qian
  • Jiayi Wu
  • Yang Zhou
  • Wentao Liu
  • Ju Ren
  • Siming Fan
  • Yaoxue Zhang

Existing research on human-centric video understanding typically focuses on analyzing specific moments or entire videos. However, many applications require higher precision at the frame level. In this work, we propose a novel task, BestShot, which aims to locate highlight frames within human-centric videos through language queries. This task requires not only a deep semantic understanding of human actions but also precise temporal localization. To support this task, we introduce the BestShot Benchmark. The benchmark is meticulously constructed by combining human-annotated highlight frames, duration labels and detailed textual descriptions. These descriptions cover three critical elements: (1) Visual content; (2) Fine-grained actions; and (3) Human pose descriptions. Together, these elements provide the necessary precision to identify the exact highlight frames in videos. To tackle this problem, we have collected two distinct datasets: (i) ShotGPT4o Dataset, which is algorithmically generated by GPT-4o and (ii) Image-SMPLText Dataset, which features large-scale and accurate per-frame pose descriptions using PoseScript and existing pose estimation datasets. Based on these datasets, we present a strong baseline model, ShotVL, fine-tuned from InternVL, specifically for BestShot. We highlight the impressive zero-shot capabilities of our model and offer comparative analyses with existing state-of-the-art (SOTA) models. ShotVL demonstrates a significant 64% improvement over InternVL on the BestShot Benchmark and a notable 68% improvement on the THUMOS14 Benchmark, while maintaining SOTA performance in general image classification and retrieval.

PDF Details DOI

NeurIPS Conference 2025 Conference Paper

The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio

  • Sicong Leng
  • Yun Xing
  • Zesen Cheng
  • Yang Zhou
  • Hang Zhang
  • Xin Li
  • Deli Zhao
  • Shijian Lu

Recent advancements in large multimodal models (LMMs) have significantly enhanced performance across diverse tasks, with ongoing efforts to further integrate additional modalities such as video and audio. However, most existing LMMs remain vulnerable to hallucinations, the discrepancy between the factual multimodal input and the generated textual output, which has limited their applicability in various real-world scenarios. This paper presents the first systematic investigation of hallucinations in LMMs involving the three most common modalities: language, visual, and audio. Our study reveals two key contributors to hallucinations: overreliance on unimodal priors and spurious inter-modality correlations. To address these challenges, we introduce the benchmark The Curse of Multi-Modalities (CMM), which comprehensively evaluates hallucinations in LMMs, providing a detailed analysis of their underlying issues. Our findings highlight key vulnerabilities, including imbalances in modality integration and biases from training data, underscoring the need for balanced cross-modal learning and enhanced hallucination mitigation strategies. Based on our observations and findings, we suggest potential research directions that could enhance the reliability of LMMs.

PDF Details

AAAI Conference 2025 Conference Paper

Treasures in Discarded Weights for LLM Quantization

  • Hao Yu
  • Yang Zhou
  • Bohua Chen
  • Zelan Yang
  • Shen Li
  • Yong Li
  • Jianxin Wu

In recent years, large language models (LLMs) have developed rapidly and revolutionized natural language processing. However, high storage overhead and computing costs limit LLM deployment in resource-constrained environments. Quantization algorithms can effectively compress LLMs and accelerate inference, but they lead to loss in precision, especially in low-bit scenarios. In this paper, we find that the discarded weight values caused by quantization in fact contain treasures to improve LLMs' accuracy. To excavate those hidden treasures, we construct search spaces around these discarded weights and those weights within the search space can seamlessly be incorporated into the original quantization weights. To determine which weights should be merged, we design a plug-and-play weight compensation framework to capture global information and keep the weights with the highest potential benefits. Our framework can be combined with various LLM quantization algorithms to achieve higher precision without additional inference overhead. We validate the effectiveness of our approach on widely used benchmark datasets for LLMs.

PDF Details DOI

IROS Conference 2025 Conference Paper

Unsupervised Anomaly Detection Improves Imitation Learning for Autonomous Racing

  • Yuang Geng
  • Yang Zhou
  • Yuyang Zhang
  • Zhongzheng Ren Zhang
  • Kang Yang
  • Tyler Ruble
  • Giancarlo Vidal
  • Ivan Ruchkin

Imitation Learning (IL) has shown significant promise in autonomous driving, but its performance heavily depends on the quality of training data. Noisy or corrupted sensor inputs can degrade learned policies, leading to unsafe behavior. This paper presents an unsupervised anomaly detection approach to automatically filter out abnormal images from driving datasets, thereby enhancing IL performance. Our method leverages a Convolutional Autoencoder with a novel latent reference loss, which forces abnormal images to reconstruct with higher errors than normal images. This enables effective anomaly detection without requiring manually labeled data. We validate our approach on the realistic DonkeyCar autonomous racing platform, demonstrating that filtering videos significantly improves IL policies, as measured by a 25-40% reduction in cross-track error. Compared to baseline and ablation models, our method achieves superior anomaly detection across three real-world video corruptions: collision-based occlusions, transparent obstructions, and raindrop interference. The results highlight the effectiveness of unsupervised video anomaly detection in improving the robustness and performance of IL-based autonomous control. Video: https://youtu.be/RjJ3nZR6RQ

Details

NeurIPS Conference 2025 Conference Paper

VividFace: A Robost and High-Fidelity Video Face Swapping Framework

  • Hao Shao
  • Shulun Wang
  • Yang Zhou
  • Guanglu Song
  • Dailan He
  • Zhuofan Zong
  • Shuo Qin
  • Yu Liu

Video face swapping has seen increasing adoption in diverse applications, yet existing methods primarily trained on static images struggle to address temporal consistency and complex real-world scenarios. To overcome these limitations, we propose the first video face swapping framework, VividFace, a robust and high-fidelity diffusion-based framework. VividFace employs a novel hybrid training strategy that leverages abundant static image data alongside temporal video sequences, enabling it to effectively model temporal coherence and identity consistency in videos. Central to our approach is a carefully designed diffusion model integrated with a specialized VAE, capable of processing image-video hybrid data efficiently. To further enhance identity and pose disentanglement, we introduce and release the Attribute-Identity Disentanglement Triplet (AIDT) dataset, comprising a large-scale collection of triplets where each set contains three face images—two sharing the same pose and two sharing the same identity. Augmented comprehensively with occlusion scenarios, AIDT significantly boosts the robustness of VividFace against occlusions. Moreover, we incorporate advanced 3D reconstruction techniques as conditioning inputs to address significant pose variations effectively. Extensive experiments demonstrate that VividFace achieves state-of-the-art performance in identity preservation, temporal consistency, and visual realism, surpassing existing methods while requiring fewer inference steps. Our framework notably mitigates common challenges such as temporal flickering, identity loss, and sensitivity to occlusions and pose variations. The AIDT dataset, source code, and pre-trained weights will be released to support future research. The code and pretrained weights are available on the project page.

PDF Details

NeurIPS Conference 2024 Conference Paper

ActAnywhere: Subject-Aware Video Background Generation

  • Boxiao Pan
  • Zhan Xu
  • Chun-Hao P. Huang
  • Krishna K. Singh
  • Yang Zhou
  • Leonidas J. Guibas
  • Jimei Yang

We study a novel problem to automatically generate video background that tailors to foreground subject motion. It is an important problem for the movie industry and visual effects community, which traditionally requires tedious manual efforts to solve. To this end, we propose ActAnywhere, a video diffusion model that takes as input a sequence of foreground subject segmentation and an image of a novel background and generates a video of the subject interacting in this background. We train our model on a large-scale dataset of 2. 4M videos of human-scene interactions. Through extensive evaluation, we show that our model produces videos with realistic foreground-background interaction while strictly following the guidance of the condition image. Our model generalizes to diverse scenarios including non-human subjects, gaming and animation clips, as well as videos with multiple moving subjects. Both quantitative and qualitative comparisons demonstrate that our model significantly outperforms existing methods, which fail to accomplish the studied task. Please visit our project webpage at https: //actanywhere. github. io.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

BenchX: A Unified Benchmark Framework for Medical Vision-Language Pretraining on Chest X-Rays

  • Yang Zhou
  • Tan L. Faith
  • Yanyu Xu
  • Sicong Leng
  • Xinxing Xu
  • Yong Liu
  • Rick S. Goh

Medical Vision-Language Pretraining (MedVLP) shows promise in learning generalizable and transferable visual representations from paired and unpaired medical images and reports. MedVLP can provide useful features to downstream tasks and facilitate adapting task-specific models to new setups using fewer examples. However, existing MedVLP methods often differ in terms of datasets, preprocessing, and finetuning implementations. This pose great challenges in evaluating how well a MedVLP method generalizes to various clinically-relevant tasks due to the lack of unified, standardized, and comprehensive benchmark. To fill this gap, we propose BenchX, a unified benchmark framework that enables head-to-head comparison and systematical analysis between MedVLP methods using public chest X-ray datasets. Specifically, BenchX is composed of three components: 1) Comprehensive datasets covering nine datasets and four medical tasks; 2) Benchmark suites to standardize data preprocessing, train-test splits, and parameter selection; 3) Unified finetuning protocols that accommodate heterogeneous MedVLP methods for consistent task adaptation in classification, segmentation, and report generation, respectively. Utilizing BenchX, we establish baselines for nine state-of-the-art MedVLP methods and found that the performance of some early MedVLP methods can be enhanced to surpass more recent ones, prompting a revisiting of the developments and conclusions from prior works in MedVLP. Our code are available at https: //github. com/yangzhou12/BenchX.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

DMesh: A Differentiable Mesh Representation

  • Sanghyun Son
  • Matheus Gadelha
  • Yang Zhou
  • Zexiang Xu
  • Ming C. Lin
  • Yi Zhou

We present a differentiable representation, DMesh, for general 3D triangular meshes. DMesh considers both the geometry and connectivity information of a mesh. In our design, we first get a set of convex tetrahedra that compactly tessellates the domain based on Weighted Delaunay Triangulation (WDT), and select triangular faces on the tetrahedra to define the final mesh. We formulate probability of faces to exist on the actual surface in a differentiable manner based on the WDT. This enables DMesh to represent meshes of various topology in a differentiable way, and allows us to reconstruct the mesh under various observations, such as point clouds and multi-view images using gradient-based optimization. We publicize the source code and supplementary material at our project page (https: //sonsang. github. io/dmesh-project).

PDF Details DOI

IJCAI Conference 2024 Conference Paper

DVPE: Divided View Position Embedding for Multi-View 3D Object Detection

  • Jiasen Wang
  • Zhenglin Li
  • Ke Sun
  • Xianyuan Liu
  • Yang Zhou

Sparse query-based paradigms have achieved significant success in multi-view 3D detection for autonomous vehicles. Current research faces challenges in balancing between enlarging receptive fields and reducing interference when aggregating multi-view features. Moreover, different poses of cameras present challenges in training global attention models. To address these problems, this paper proposes a divided view method, in which features are modeled globally via the visibility cross-attention mechanism, but interact only with partial features in a divided local virtual space. This effectively reduces interference from other irrelevant features and alleviates the training difficulties of the transformer by decoupling the position embedding from camera poses. Additionally, 2D historical RoI features are incorporated into the object-centric temporal modeling to utilize high-level visual semantic information. The model is trained using a one-to-many assignment strategy to facilitate stability. Our framework, named DVPE, achieves state-of-the-art performance (57. 2% mAP and 64. 5% NDS) on the nuScenes test set. Codes will be available at https: //github. com/dop0/DVPE.

PDF Details DOI

TIST Journal 2024 Journal Article

Efficient Federated Learning Using Dynamic Update and Adaptive Pruning with Momentum on Shared Server Data

  • Ji Liu
  • Juncheng Jia
  • Hong Zhang
  • Yuhui Yun
  • Leye Wang
  • Yang Zhou
  • Huaiyu Dai
  • Dejing Dou

Despite achieving remarkable performance, Federated Learning (FL) encounters two important problems, i.e., low training efficiency and limited computational resources. In this article, we propose a new FL framework, i.e., FedDUMAP, with three original contributions, to leverage the shared insensitive data on the server in addition to the distributed data in edge devices so as to efficiently train a global model. First, we propose a simple dynamic server update algorithm, which takes advantage of the shared insensitive data on the server while dynamically adjusting the update steps on the server in order to speed up the convergence and improve the accuracy. Second, we propose an adaptive optimization method with the dynamic server update algorithm to exploit the global momentum on the server and each local device for superior accuracy. Third, we develop a layer-adaptive model pruning method to carry out specific pruning operations, which is adapted to the diverse features of each layer so as to attain an excellent tradeoff between effectiveness and efficiency. Our proposed FL model, FedDUMAP, combines the three original techniques and has a significantly better performance compared with baseline approaches in terms of efficiency (up to 16.9 times faster), accuracy (up to 20.4% higher), and computational cost (up to 62.6% smaller).

Details DOI

IJCAI Conference 2024 Conference Paper

EvaNet: Elevation-Guided Flood Extent Mapping on Earth Imagery

  • Mirza Tanzim Sami
  • Da Yan
  • Saugat Adhikari
  • Lyuheng Yuan
  • Jiao Han
  • Zhe Jiang
  • Jalal Khalil
  • Yang Zhou

Accurate and timely mapping of flood extent from high resolution satellite imagery plays a crucial role in disaster management such as damage assessment and relief activities. However, current state-of-the-art solutions are based on U-Net, which cannot segment the flood pixels accurately due to the ambiguous pixels (e. g. , tree canopies, clouds) that prevent a direct judgement from only the spectral features. Thanks to the digital elevation model (DEM) data readily available from sources such as United States Geological Survey (USGS), this work explores the use of an elevation map to improve flood extent mapping. We propose, EvaNet, an elevation-guided segmentation model based on the encoder-decoder architecture with two novel techniques: (1) a loss function encoding the physical law of gravity that if a location is flooded (resp. dry), then its adjacent locations with a lower (resp. higher) elevation must also be flooded (resp. dry); (2) a new (de)convolution operation that integrates the elevation map by a location-sensitive gating mechanism to regulate how much spectral features flow through adjacent layers. Extensive experiments show that EvaNet significantly outperforms the U-Net baselines, and works as a perfect drop-in replacement for U-Net in existing solutions to flood extent mapping. EvaNet is open-sourced at https: //github. com/MTSami/EvaNet.

PDF Details DOI

AAAI Conference 2024 Conference Paper

FedASMU: Efficient Asynchronous Federated Learning with Dynamic Staleness-Aware Model Update

  • Ji Liu
  • Juncheng Jia
  • Tianshi Che
  • Chao Huo
  • Jiaxiang Ren
  • Yang Zhou
  • Huaiyu Dai
  • Dejing Dou

As a promising approach to deal with distributed data, Federated Learning (FL) achieves major advancements in recent years. FL enables collaborative model training by exploiting the raw data dispersed in multiple edge devices. However, the data is generally non-independent and identically distributed, i.e., statistical heterogeneity, and the edge devices significantly differ in terms of both computation and communication capacity, i.e., system heterogeneity. The statistical heterogeneity leads to severe accuracy degradation while the system heterogeneity significantly prolongs the training process. In order to address the heterogeneity issue, we propose an Asynchronous Staleness-aware Model Update FL framework, i.e., FedASMU, with two novel methods. First, we propose an asynchronous FL system model with a dynamical model aggregation method between updated local models and the global model on the server for superior accuracy and high efficiency. Then, we propose an adaptive local model adjustment method by aggregating the fresh global model with local models on devices to further improve the accuracy. Extensive experimentation with 6 models and 5 public datasets demonstrates that FedASMU significantly outperforms baseline approaches in terms of accuracy (0.60% to 23.90% higher) and efficiency (3.54% to 97.98% faster).

PDF Details DOI

ICML Conference 2024 Conference Paper

HexGen: Generative Inference of Large Language Model over Heterogeneous Environment

  • Youhe Jiang
  • Ran Yan
  • Xiaozhe Yao
  • Yang Zhou
  • Beidi Chen
  • Binhang Yuan

Serving generative inference of the large language model is a crucial component of contemporary AI applications. In this paper, our focus lies in deploying such services in a heterogeneous and cross-datacenter setting to mitigate the substantial inference costs typically associated with a single centralized datacenter. Towards this end, we propose HexGen, a flexible distributed inference engine that uniquely supports the asymmetric partition of generative inference computations over both tensor model parallelism and pipeline parallelism, which allows for effective deployment across diverse GPUs interconnected by a fully heterogeneous network. We further propose a sophisticated scheduling algorithm grounded in constrained optimization that can adaptively assign asymmetric inference computation across the GPUs to fulfill inference requests while maintaining acceptable latency levels. We conduct an extensive empirical study to evaluate the efficiency of HexGen by serving the state-of-the-art Llama-2 (70B) model. The experimental results suggest that HexGen can choose to achieve up to $2. 3\times$ lower latency deadlines or tolerate up to $4\times$ more traffic request rates compared with the homogeneous baseline given the same budget.

Details

AAAI Conference 2024 Conference Paper

In-Hand 3D Object Reconstruction from a Monocular RGB Video

  • Shijian Jiang
  • Qi Ye
  • Rengan Xie
  • Yuchi Huo
  • Xiang Li
  • Yang Zhou
  • Jiming Chen

Our work aims to reconstruct a 3D object that is held and rotated by a hand in front of a static RGB camera. Previous methods that use implicit neural representations to recover the geometry of a generic hand-held object from multi-view images achieved compelling results in the visible part of the object. However, these methods falter in accurately capturing the shape within the hand-object contact region due to occlusion. In this paper, we propose a novel method that deals with surface reconstruction under occlusion by incorporating priors of 2D occlusion elucidation and physical contact constraints. For the former, we introduce an object amodal completion network to infer the 2D complete mask of objects under occlusion. To ensure the accuracy and view consistency of the predicted 2D amodal masks, we devise a joint optimization method for both amodal mask refinement and 3D reconstruction. For the latter, we impose penetration and attraction constraints on the local geometry in contact regions. We evaluate our approach on HO3D and HOD datasets and demonstrate that it outperforms the state-of-the-art methods in terms of reconstruction surface quality, with an improvement of 52% on HO3D and 20% on HOD. Project webpage: https://east-j.github.io/ihor.

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

SIRIUS : Contexual Sparisty with Correction for Efficient LLMs

  • Yang Zhou
  • Zhuoming Chen
  • Zhaozhuo Xu
  • Xi V. Lin
  • Beidi Chen

With the blossom of large language models (LLM), inference efficiency becomes increasingly important. Various approximate methods are proposed to reduce the cost at inference time. Contextual Sparsity (CS) is appealing for its training-free nature and its ability to reach a higher compression ratio seemingly without significant performance degradation. However, after a comprehensive evaluation of contextual sparsity methods on various complex generation tasks, we find that although CS succeeds in prompt-understanding tasks, it significantly degrades the model performance for reasoning, deduction, and knowledge-based tasks. Despite the gap in end-to-end accuracy, we observed that sparse models and original models often share the general problem-solving logic and require only a few token corrections to recover the original model performance. This paper introduces SIRIUS, an efficient correction mechanism, which significantly boosts CS models on reasoning tasks while maintaining its efficiency gain. SIRIUS is evaluated on 6 models with 8 difficult generation tasks in reasoning, deduction, and coding and shows consistent effectiveness and efficiency. Also, we carefully develop a system implementation for SIRIUS and show that SIRIUS delivers theoretical latency reduction with roughly a 20% reduction in latency for 8B model on-chip and a 35% reduction in latency for 70B model offloading. We open-source our implementation of Sirius at https: //github. com/Infini-AI-Lab/Sirius. git.

PDF Details DOI

AAAI Conference 2024 Conference Paper

Spatial-Logic-Aware Weakly Supervised Learning for Flood Mapping on Earth Imagery

  • Zelin Xu
  • Tingsong Xiao
  • Wenchong He
  • Yu Wang
  • Zhe Jiang
  • Shigang Chen
  • Yiqun Xie
  • Xiaowei Jia

Flood mapping on Earth imagery is crucial for disaster management, but its efficacy is hampered by the lack of high-quality training labels. Given high-resolution Earth imagery with coarse and noisy training labels, a base deep neural network model, and a spatial knowledge base with label constraints, our problem is to infer the true high-resolution labels while training neural network parameters. Traditional methods are largely based on specific physical properties and thus fall short of capturing the rich domain constraints expressed by symbolic logic. Neural-symbolic models can capture rich domain knowledge, but existing methods do not address the unique spatial challenges inherent in flood mapping on high-resolution imagery. To fill this gap, we propose a spatial-logic-aware weakly supervised learning framework. Our framework integrates symbolic spatial logic inference into probabilistic learning in a weakly supervised setting. To reduce the time costs of logic inference on vast high-resolution pixels, we propose a multi-resolution spatial reasoning algorithm to infer true labels while training neural network parameters. Evaluations of real-world flood datasets show that our model outperforms several baselines in prediction accuracy. The code is available at https://github.com/spatialdatasciencegroup/SLWSL.

PDF Details DOI

ICRA Conference 2024 Conference Paper

TPGP: Temporal-Parametric Optimization with Deep Grasp Prior for Dexterous Motion Planning

  • Haoming Li 0004
  • Qi Ye 0001
  • Yuchi Huo
  • Qingtao Liu
  • Shijian Jiang
  • Tao Zhou
  • Xiang Li
  • Yang Zhou

Grasping motion planning aims to find a feasible grasping trajectory in the configuration space given an input target grasp. While optimizing grasp motion with two or three-fingered grippers has been well studied, the study on natural grasp motion planning with a dexterous hand remains a very challenging problem due to the high dimensional working space. In this work, we propose a novel temporal-parametric grasp prior (TPGP) optimization method to simplify the difficulty of grasping trajectory optimization for the dexterous hand while maintaining smooth and natural properties of the grasping motion. Specifically, we formulate the discrete trajectory parameters into a temporal-based parameterization, where the prior constraint provided by a hand poser network, is introduced to ensure that hand pose is natural and reasonable throughout the trajectory. Finally, we present a joint target optimization strategy to enhance the target pose for more feasible trajectories. Extensive validations on two public datasets show that our method outperforms state-of-the-art methods regarding grasp motion on various metrics.

Details

IJCAI Conference 2023 Conference Paper

Contact2Grasp: 3D Grasp Synthesis via Hand-Object Contact Constraint

  • Haoming Li
  • Xinzhuo Lin
  • Yang Zhou
  • Xiang Li
  • Yuchi Huo
  • Jiming Chen
  • Qi Ye

3D grasp synthesis generates grasping poses given an input object. Existing works tackle the problem by learning a direct mapping from objects to the distributions of grasping poses. However, because the physical contact is sensitive to small changes in pose, the high-nonlinear mapping between 3D object representation to valid poses is considerably non-smooth, leading to poor generation efficiency and restricted generality. To tackle the challenge, we introduce an intermediate variable for grasp contact areas to constrain the grasp generation; in other words, we factorize the mapping into two sequential stages by assuming that grasping poses are fully constrained given contact maps: 1) we first learn contact map distributions to generate the potential contact maps for grasps; 2) then learn a mapping from the contact maps to the grasping poses. Further, we propose a penetration-aware optimization with the generated contacts as a consistency constraint for grasp refinement. Extensive validations on two public datasets show that our method outperforms state-of-the-art methods regarding grasp generation on various metrics.

PDF Details DOI

TIST Journal 2022 Journal Article

Earth Imagery Segmentation on Terrain Surface with Limited Training Labels: A Semi-supervised Approach based on Physics-Guided Graph Co-Training

  • Wenchong He
  • Arpan Man Sainju
  • Zhe Jiang
  • Da Yan
  • Yang Zhou

Given earth imagery with spectral features on a terrain surface, this paper studies surface segmentation based on both explanatory features and surface topology. The problem is important in many spatial and spatiotemporal applications such as flood extent mapping in hydrology. The problem is uniquely challenging for several reasons: first, the size of earth imagery on a terrain surface is often much larger than the input of popular deep convolutional neural networks; second, there exists topological structure dependency between pixel classes on the surface, and such dependency can follow an unknown and non-linear distribution; third, there are often limited training labels. Existing methods for earth imagery segmentation often divide the imagery into patches and consider the elevation as an additional feature channel. These methods do not fully incorporate the spatial topological structural constraint within and across surface patches and thus often show poor results, especially when training labels are limited. Existing methods on semi-supervised and unsupervised learning for earth imagery often focus on learning representation without explicitly incorporating surface topology. In contrast, we propose a novel framework that explicitly models the topological skeleton of a terrain surface with a contour tree from computational topology, which is guided by the physical constraint (e.g., water flow direction on terrains). Our framework consists of two neural networks: a convolutional neural network (CNN) to learn spatial contextual features on a 2D image grid, and a graph neural network (GNN) to learn the statistical distribution of physics-guided spatial topological dependency on the contour tree. The two models are co-trained via variational EM. Evaluations on the real-world flood mapping datasets show that the proposed models outperform baseline methods in classification accuracy, especially when training labels are limited.

Details DOI

AAAI Conference 2022 Conference Paper

Efficient Device Scheduling with Multi-Job Federated Learning

  • Chendi Zhou
  • Ji Liu
  • Juncheng Jia
  • Jingbo Zhou
  • Yang Zhou
  • Huaiyu Dai
  • Dejing Dou

Recent years have witnessed a large amount of decentralized data in multiple (edge) devices of end-users, while the aggregation of the decentralized data remains difficult for machine learning jobs due to laws or regulations. Federated Learning (FL) emerges as an effective approach to handling decentralized data without sharing the sensitive raw data, while collaboratively training global machine learning models. The servers in FL need to select (and schedule) devices during the training process. However, the scheduling of devices for multiple jobs with FL remains a critical and open problem. In this paper, we propose a novel multi-job FL framework to enable the parallel training process of multiple jobs. The framework consists of a system model and two scheduling methods. In the system model, we propose a parallel training process of multiple jobs, and construct a cost model based on the training time and the data fairness of various devices during the training process of diverse jobs. We propose a reinforcement learning-based method and a Bayesian optimization-based method to schedule devices for multiple jobs while minimizing the cost. We conduct extensive experimentation with multiple jobs and datasets. The experimental results show that our proposed approaches significantly outperform baseline approaches in terms of training time (up to 8. 67 times faster) and accuracy (up to 44. 6% higher).

PDF Details

IJCAI Conference 2022 Conference Paper

FedDUAP: Federated Learning with Dynamic Update and Adaptive Pruning Using Shared Data on the Server

  • Hong Zhang
  • Ji Liu
  • Juncheng Jia
  • Yang Zhou
  • Huaiyu Dai
  • Dejing Dou

Despite achieving remarkable performance, Federated Learning (FL) suffers from two critical challenges, i. e. , limited computational resources and low training efficiency. In this paper, we propose a novel FL framework, i. e. , FedDUAP, with two original contributions, to exploit the insensitive data on the server and the decentralized data in edge devices to further improve the training efficiency. First, a dynamic server update algorithm is designed to exploit the insensitive data on the server, in order to dynamically determine the optimal steps of the server update for improving the convergence and accuracy of the global model. Second, a layer-adaptive model pruning method is developed to perform unique pruning operations adapted to the different dimensions and importance of multiple layers, to achieve a good balance between efficiency and effectiveness. By integrating the two original techniques together, our proposed FL model, FedDUAP, significantly outperforms baseline approaches in terms of accuracy (up to 4. 8% higher), efficiency (up to 2. 8 times faster), and computational cost (up to 61. 9% smaller).

PDF Details DOI

IJCAI Conference 2022 Conference Paper

None Class Ranking Loss for Document-Level Relation Extraction

  • Yang Zhou
  • Wee Sun Lee

Document-level relation extraction (RE) aims at extracting relations among entities expressed across multiple sentences, which can be viewed as a multi-label classification problem. In a typical document, most entity pairs do not express any pre-defined relation and are labeled as "none" or "no relation". For good document-level RE performance, it is crucial to distinguish such none class instances (entity pairs) from those of pre-defined classes (relations). However, most existing methods only estimate the probability of pre-defined relations independently without considering the probability of "no relation". This ignores the context of entity pairs and the label correlations between the none class and pre-defined classes, leading to sub-optimal predictions. To address this problem, we propose a new multi-label loss that encourages large margins of label confidence scores between each pre-defined class and the none class, which enables captured label correlations and context-dependent thresholding for label prediction. To gain further robustness against positive-negative imbalance and mislabeled data that could appear in real-world RE datasets, we propose a margin regularization and a margin shifting technique. Experimental results demonstrate that our method significantly outperforms existing multi-label losses for document-level RE and works well in other multi-label tasks such as emotion classification when none class instances are available for training.

PDF Details DOI

JMLR Journal 2022 Journal Article

Nonparametric Principal Subspace Regression

  • Yang Zhou
  • Mark Koudstaal
  • Dengdeng Yu
  • Dehan Kong
  • Fang Yao

In scientific applications, multivariate observations often come in tandem with temporal or spatial covariates, with which the underlying signals vary smoothly. The standard approaches such as principal component analysis and factor analysis neglect the smoothness of the data, while multivariate linear or nonparametric regression fails to leverage the correlation information among multivariate response variables. We propose a novel approach named nonparametric principal subspace regression to overcome these issues. By decoupling the model discrepancy, a simple two-step estimation procedure is introduced, which takes advantage of the low-rank approximation while keeping smooth dynamics. The theoretical property of the proposed procedure is established under an increasing-dimension framework. We demonstrate the favorable performance of our method in comparison with its counterpart, the conventional nonparametric regression, from both theoretical and numerical perspectives. [abs] [ pdf ][ bib ] &copy JMLR 2022. ( edit, beta )

PDF Details

NeurIPS Conference 2022 Conference Paper

Prompt Certified Machine Unlearning with Randomized Gradient Smoothing and Quantization

  • Zijie Zhang
  • Yang Zhou
  • Xin Zhao
  • Tianshi Che
  • Lingjuan Lyu

The right to be forgotten calls for efficient machine unlearning techniques that make trained machine learning models forget a cohort of data. The combination of training and unlearning operations in traditional machine unlearning methods often leads to the expensive computational cost on large-scale data. This paper presents a prompt certified machine unlearning algorithm, PCMU, which executes one-time operation of simultaneous training and unlearning in advance for a series of machine unlearning requests, without the knowledge of the removed/forgotten data. First, we establish a connection between randomized smoothing for certified robustness on classification and randomized smoothing for certified machine unlearning on gradient quantization. Second, we propose a prompt certified machine unlearning model based on randomized data smoothing and gradient quantization. We theoretically derive the certified radius R regarding the data change before and after data removals and the certified budget of data removals about R. Last but not least, we present another practical framework of randomized gradient smoothing and quantization, due to the dilemma of producing high confidence certificates in the first framework. We theoretically demonstrate the certified radius R' regarding the gradient change, the correlation between two types of certified radii, and the certified budget of data removals about R'.

PDF Details

TCS Journal 2022 Journal Article

Streaming algorithms for monotone non-submodular function maximization under a knapsack constraint on the integer lattice

  • Jingjing Tan
  • Fengmin Wang
  • Weina Ye
  • Xiaoqing Zhang
  • Yang Zhou

The study of non-submodular maximization on the integer lattice is an important extension of submodular optimization. In this paper, streaming algorithms for maximizing non-negative monotone non-submodular functions with knapsack constraint on integer lattice are considered. We first design a two-pass StreamingKnapsack algorithm combining with BinarySearch as a subroutine for this problem. By introducing the DR ratio γ and the weak DR ratio γ w of the non-submodular objective function, we obtain that the approximation ratio is min ⁡ { γ 2 ( 1 − ε ) / 2 γ + 1, 1 − 1 / γ w 2 γ − ε }, the total memory complexity is O ( K log ⁡ K / ε ), and the total query complexity for each element is O ( log ⁡ K log ⁡ ( K / ε 2 ) / ε ). Then, we design a one-pass streaming algorithm by dynamically updating the maximal function value among unit vectors along with the currently arriving element. Finally, in order to decrease the memory complexity, we design an improved StreamingKnapsack algorithm and reduce the memory complexity to O ( K / ε 2 ).

Details DOI

TCS Journal 2022 Journal Article

The submodularity of two-stage stochastic maximum-weight independent set problems

  • Min Li
  • Hao Xiao
  • Qian Liu
  • Yang Zhou

In this paper, we extend the maximal independent set problem to two-stage stochastic case: given an independence system associated with one deterministic weight function and a random weight function, the goal is to find two nonoverlapping independent subsets from these two stages with the maximum total weight. In this paper, we study the submodularity of three kinds of two-stage independent set problems with max-weight. When the independent set problem is a matroid constraint, we can show its submodularity. However, neither submodular nor supermodular maximization problem can be obtained for the knapsack independent set problem by designing a counterexample. At last, we show that the robust two-stage stochastic maximum-weight uniform matroid problem can be formulated as a γ-submodular problem with cardinality constraint and also give a lower bound for γ.

Details DOI

TCS Journal 2021 Journal Article

Approximation algorithms for fuzzy C-means problem based on seeding method

  • Qian Liu
  • Jianxin Liu
  • Min Li
  • Yang Zhou

As a kind of important soft clustering model, the fuzzy C-means method is widely applied in many fields. In this method, instead of the strict distributive ability in the classical k-means method, all the sample points are endowed with degrees of membership to each center to depict the fuzzy clustering. In this paper, we show that the fuzzy C-means++ algorithm, which introduces the k-means++ algorithm as a seeding strategy, gives a solution for which the approximation guarantee is O ( k 2 ln ⁡ k ). A novel seeding algorithm is then designed based on the contribution of the fuzzy potential function, which improves the approximation ratio to O ( k ln ⁡ k ). Preliminary numerical experiments are proposed to support the theoretical results of this paper.

Details DOI

AAAI Conference 2021 System Paper

CogNet: Bridging Linguistic Knowledge, World Knowledge and Commonsense Knowledge

  • Chenhao Wang
  • Yubo Chen
  • Zhipeng Xue
  • Yang Zhou
  • Jun Zhao

In this paper, we present CogNet, a knowledge base (KB) dedicated to integrating three types of knowledge: (1) linguistic knowledge from FrameNet, which schematically describes situations, objects and events. (2) world knowledge from YAGO, Freebase, DBpedia and Wikidata, which provides explicit knowledge about specific instances. (3) commonsense knowledge from ConceptNet, which describes implicit general facts. To model these different types of knowledge consistently, we introduce a three-level unified frame-styled representation architecture. To integrate free-form commonsense knowledge with other structured knowledge, we propose a strategy that combines automated labeling and crowdsourced annotation. At present, CogNet integrates 1, 000+ semantic frames from linguistic KBs, 20, 000, 000+ frame instances from world KBs, as well as 90, 000+ commonsense assertions from commonsense KBs. All these data can be easily queried and explored on our online platform, and free to download in RDF format for utilization under a CC-BY-SA 4. 0 license. The demo and data are available at http: //cognet. top/.

PDF Details

IJCAI Conference 2021 Conference Paper

Knowledge-Aware Dialogue Generation via Hierarchical Infobox Accessing and Infobox-Dialogue Interaction Graph Network

  • Sixing Wu
  • Minghui Wang
  • Dawei Zhang
  • Yang Zhou
  • Ying Li
  • Zhonghai Wu

Due to limited knowledge carried by queries, traditional dialogue systems often face the dilemma of generating boring responses, leading to poor user experience. To alleviate this issue, this paper proposes a novel infobox knowledge-aware dialogue generation approach, HITA-Graph, with three unique features. First, open-domain infobox tables that describe entities with relevant attributes are adopted as the knowledge source. An order-irrelevance Hierarchical Infobox Table Encoder is proposed to represent an infobox table at three levels of granularity. In addition, an Infobox-Dialogue Interaction Graph Network is built to effectively integrate the infobox context and the dialogue context into a unified infobox representation. Second, a Hierarchical Infobox Attribute Attention mechanism is developed to access the encoded infobox knowledge at different levels of granularity. Last but not least, a Dynamic Mode Fusion strategy is designed to allow the Decoder to select a vocabulary word or copy a word from the given infobox/query. We extract infobox tables from Chinese Wikipedia and construct an infobox knowledge base. Extensive evaluation on an open-released Chinese corpus demonstrates the superior performance of our approach against several representative methods.

PDF Details DOI

NeurIPS Conference 2021 Conference Paper

Validating the Lottery Ticket Hypothesis with Inertial Manifold Theory

  • Zeru Zhang
  • Jiayin Jin
  • Zijie Zhang
  • Yang Zhou
  • Xin Zhao
  • Jiaxiang Ren
  • Ji Liu
  • Lingfei Wu

Despite achieving remarkable efficiency, traditional network pruning techniques often follow manually-crafted heuristics to generate pruned sparse networks. Such heuristic pruning strategies are hard to guarantee that the pruned networks achieve test accuracy comparable to the original dense ones. Recent works have empirically identified and verified the Lottery Ticket Hypothesis (LTH): a randomly-initialized dense neural network contains an extremely sparse subnetwork, which can be trained to achieve similar accuracy to the former. Due to the lack of theoretical evidence, they often need to run multiple rounds of expensive training and pruning over the original large networks to discover the sparse subnetworks with low accuracy loss. By leveraging dynamical systems theory and inertial manifold theory, this work theoretically verifies the validity of the LTH. We explore the possibility of theoretically lossless pruning as well as one-time pruning, compared with existing neural network pruning and LTH techniques. We reformulate the neural network optimization problem as a gradient dynamical system and reduce this high-dimensional system onto inertial manifolds to obtain a low-dimensional system regarding pruned subnetworks. We demonstrate the precondition and existence of pruned subnetworks and prune the original networks in terms of the gap in their spectrum that make the subnetworks have the smallest dimensions.

PDF Details

AAAI Conference 2021 Conference Paper

What the Role is vs. What Plays the Role: Semi-Supervised Event Argument Extraction via Dual Question Answering

  • Yang Zhou
  • Yubo Chen
  • Jun Zhao
  • Yin Wu
  • Jiexin Xu
  • Jinlong Li

Event argument extraction is an essential task in event extraction, and become particularly challenging in the case of low-resource scenarios. We solve the issues in existing studies under low-resource situations from two sides. From the perspective of the model, the existing methods always suffer from the concern of insufficient parameter sharing and do not consider the semantics of roles, which is not conducive to dealing with sparse data. And from the perspective of the data, most existing methods focus on data generation and data augmentation. However, these methods rely heavily on external resources, which is more laborious to create than obtain unlabeled data. In this paper, we propose DualQA, a novel framework, which models the event argument extraction task as question answering to alleviate the problem of data sparseness and leverage the duality of event argument recognition which is to ask “What plays the role”, as well as event role recognition which is to ask “What the role is”, to mutually improve each other. Experimental results on two datasets prove the effectiveness of our approach, especially in extremely low-resource situations.

PDF Details

NeurIPS Conference 2020 Conference Paper

Adversarial Attacks on Deep Graph Matching

  • Zijie Zhang
  • Zeru Zhang
  • Yang Zhou
  • Yelong Shen
  • Ruoming Jin
  • Dejing Dou

Despite achieving remarkable performance, deep graph learning models, such as node classification and network embedding, suffer from harassment caused by small adversarial perturbations. However, the vulnerability analysis of graph matching under adversarial attacks has not been fully investigated yet. This paper proposes an adversarial attack model with two novel attack techniques to perturb the graph structure and degrade the quality of deep graph matching: (1) a kernel density estimation approach is utilized to estimate and maximize node densities to derive imperceptible perturbations, by pushing attacked nodes to dense regions in two graphs, such that they are indistinguishable from many neighbors; and (2) a meta learning-based projected gradient descent method is developed to well choose attack starting points and to improve the search performance for producing effective perturbations. We evaluate the effectiveness of the attack model on real datasets and validate that the attacks can be transferable to other graph learning models.

PDF Details

AAAI Conference 2020 Conference Paper

EFANet: Exchangeable Feature Alignment Network for Arbitrary Style Transfer

  • Zhijie Wu
  • Chunjin Song
  • Yang Zhou
  • Minglun Gong
  • Hui Huang

Style transfer has been an important topic both in computer vision and graphics. Since the seminal work of Gatys et al. first demonstrates the power of stylization through optimization in the deep feature space, quite a few approaches have achieved real-time arbitrary style transfer with straightforward statistic matching techniques. In this work, our key observation is that only considering features in the input style image for the global deep feature statistic matching or local patch swap may not always ensure a satisfactory style transfer; see e. g. , Figure 1. Instead, we propose a novel transfer framework, EFANet, that aims to jointly analyze and better align exchangeable features extracted from the content and style image pair. In this way, the style feature from the style image seeks for the best compatibility with the content information in the content image, leading to more structured stylization results. In addition, a new whitening loss is developed for purifying the computed content features and better fusion with styles in feature space. Qualitative and quantitative experiments demonstrate the advantages of our approach.

PDF Details

IJCAI Conference 2020 Conference Paper

TopicKA: Generating Commonsense Knowledge-Aware Dialogue Responses Towards the Recommended Topic Fact

  • Sixing Wu
  • Ying Li
  • Dawei Zhang
  • Yang Zhou
  • Zhonghai Wu

Insufficient semantic understanding of dialogue always leads to the appearance of generic responses, in generative dialogue systems. Recently, high-quality knowledge bases have been introduced to enhance dialogue understanding, as well as to reduce the prevalence of boring responses. Although such knowledge-aware approaches have shown tremendous potential, they always utilize the knowledge in a black-box fashion. As a result, the generation process is somewhat uncontrollable, and it is also not interpretable. In this paper, we introduce a topic fact-based commonsense knowledge-aware approach, TopicKA. Different from previous works, TopicKA generates responses conditioned not only on the query message but also on a topic fact with an explicit semantic meaning, which also controls the direction of generation. Topic facts are recommended by a recommendation network trained under the Teacher-Student framework. To integrate the recommendation network and the generation network, this paper designs four schemes, which include two non-sampling schemes and two sampling methods. We collected and constructed a large-scale Chinese commonsense knowledge graph. Experimental results on an open Chinese benchmark dataset indicate that our model outperforms baselines in terms of both the objective and the subjective metrics.

PDF Details DOI

IROS Conference 2019 Conference Paper

Adaptive Vision-Based Control for Rope-Climbing Robot Manipulator

  • Guangli Sun
  • Xiang Li 0009
  • Peng Li 0019
  • Linzhu Yue
  • Zhen Yu
  • Yang Zhou
  • Yun-Hui Liu 0001

While the mechanism of Rope-Climbing provides much flexibility, it opens up challenges to the development of the controller for Robotic Manipulator installed on Rope-Climbing robot(RCR), which is called Rope-Climbing Robot Manipulator(RCRM) here. In particular, the deformable nature of the rope results in the vibration to the manipulator and hence affects the positioning of the end effector. In this paper, a new adaptive vision-based controller is proposed for RCRM, which enables the robot to carry out the high-accuracy task under the unknown vibration from the rope. The proposed controller guarantees the performance of the robot in twofold. First, the control problem is directly formulated in the image space such that the exact spatial relationship between the moving base of the manipulator (due to the vibrating rope) and the target (e. g. the wall) is not required. Second, novel adaptation laws are developed to estimate the vibration from the rope online and are cancelled out in the robot control input to stabilize the end effector. The stability of the closed-loop system is rigorously proved with Lyapunov methods, and experimental results are presented to illustrate the performance of the proposed controller.

Details

NeurIPS Conference 2019 Conference Paper

ETNet: Error Transition Network for Arbitrary Style Transfer

  • Chunjin Song
  • Zhijie Wu
  • Yang Zhou
  • Minglun Gong
  • Hui Huang

Numerous valuable efforts have been devoted to achieving arbitrary style transfer since the seminal work of Gatys et al. However, existing state-of-the-art approaches often generate insufficiently stylized results under challenging cases. We believe a fundamental reason is that these approaches try to generate the stylized result in a single shot and hence fail to fully satisfy the constraints on semantic structures in the content images and style patterns in the style images. Inspired by the works on error-correction, instead, we propose a self-correcting model to predict what is wrong with the current stylization and refine it accordingly in an iterative manner. For each refinement, we transit the error features across both the spatial and scale domain and invert the processed features into a residual image, with a network we call Error Transition Network (ETNet). The proposed model improves over the state-of-the-art methods with better semantic structures and more adaptive style pattern details. Various qualitative and quantitative experiments show that the key concept of both progressive strategy and error-correction leads to better results. Code and models are available at https: //github. com/zhijieW94/ETNet.

PDF Details

IROS Conference 2019 Conference Paper

Global Vision-Based Impedance Control for Robotic Wall Polishing

  • Yang Zhou
  • Xiang Li 0009
  • Linzhu Yue
  • Linhai Gui
  • Guangli Sun
  • Xin Jiang 0001
  • Yun-Hui Liu 0001

Wall polishing is a typical and essential procedure in the interior renovation. However, such works are mainly carried out by humans, which have the disadvantages of low efficiency, inconsistent quality, and issues of safety and health. A new vision-based impedance controller is proposed for polishing robots to automate the labor-intensive works. The desired impedance model is specified as the control objective to regulate the dynamic relationship between the interaction force and the motion of the robot end effector, where the motion is measured with the vision feedback. The use of the vision feedback guarantees the performance of the robot from two aspect. First, the vision feedback from the high-resolution camera ensures the accuracy of measurement of the robot end effector and hence guarantees the quality of polishing. Second, the concept of image moment is introduced such that the image Jacobian matrix is non-singular in a global sense, which guarantees the large working range of the robot. The dynamic stability of the closed-loop system is rigorously proved with Lyapunov methods, and experimental results are presented to illustrate the performance of the proposed controller.

Details

ICRA Conference 2018 Conference Paper

A Synchronization Scheme for Position Control of Multiple Rope-Climbing Robots

  • Guangli Sun
  • Xiang Li 0009
  • Peng Li 0019
  • Yu Meng
  • Yang Zhou
  • Enzhi Xu
  • Yun-Hui Liu 0001

The ability of rope-climbing robots in aloft operation is limited by its self-supporting and locomotion ability. In many applications, a given task is also too complex to be achieved by a single rope-climbing robot acting alone. The solution of multiple rope-climbing robots can overcome the limitations. However, existing control methods for rope-climbing robots are limited to single robot, and the open issue of coordination between multiple rope-climbing robots has not been systematically addressed. This paper presents a new synchronization scheme for position control of multiple rope-climbing robots, such that each robot moves to the corresponding desired position while synchronizing the heights between each other. Maintaining the same height is very important to guarantee the stability of the task-oriented manipulator installed among multiple robots, when it is performing the manipulation task. The development of the proposed controller is based on the singular perturbation approach, by treating the fast actuator dynamics as a perturbation of the slow robot dynamics, such that the lowest control complexity is achieved. The exponential stability of the overall system that consists of the fast and slow subsystems is proved by using Tikhonov’ s theorem. Experimental results are presented to illustrate the performance of the proposed controller.

Details

AAAI Conference 2017 Conference Paper

Bilinear Probabilistic Canonical Correlation Analysis via Hybrid Concatenations

  • Yang Zhou
  • Haiping Lu
  • Yiu-ming Cheung

Canonical Correlation Analysis (CCA) is a classical technique for two-view correlation analysis, while Probabilistic CCA (PCCA) provides a generative and more general viewpoint for this task. Recently, PCCA has been extended to bilinear cases for dealing with two-view matrices in order to preserve and exploit the matrix structures in PCCA. However, existing bilinear PCCAs impose restrictive model assumptions for matrix structure preservation, sacrificing generative correctness or model flexibility. To overcome these drawbacks, we propose BPCCA, a new bilinear extension of PCCA, by introducing a hybrid joint model. Our new model preserves matrix structures indirectly via hybrid vector-based and matrix-based concatenations. This enables BPCCA to gain more model flexibility in capturing two-view correlations and obtain close-form solutions in parameter estimation. Experimental results on two real-world applications demonstrate the superior performance of BPCCA over competing methods.

PDF Details

IROS Conference 2017 Conference Paper

CSMA/CA-based electrocommunication system design for underwater robot groups

  • Han Zhang 0044
  • Wei Wang 0078
  • Yang Zhou
  • Chen Wang 0005
  • Ruifeng Fan
  • Guangming Xie

Underwater communication is particularly challenging for small submarine robots that have limited power and size constraints. Inspired by weakly electric fish, a novel electric current communication (termed electrocommunication) system has been developed for small underwater robots in our previous studies. However, collision problems sometimes occur during multiple robots communication because the electrocommunication network shares a common channel. In this paper, a new CSMA/CA-based electrocommunication system is presented to solve this collision problem. An efficient circuit for communication channel state (busy or idle) detection is proposed. After that, a compact Carrier Sense Multiple Access with Collision Avoidance (CSMA/CA) algorithm is introduced and finally implemented on the system to judiciously avoid collision during multiple robot communication. The effectiveness of the proposed CSMA/CA-based communication system for underwater robot groups is well verified by both simulations and experiments with three robotic fish models.

Details

TIST Journal 2016 Journal Article

CSM

  • Yexi Jiang
  • Chang-Shing Perng
  • Anca Sailer
  • Ignacio Silva-Lepe
  • Yang Zhou
  • Tao Li

The cloud service marketplace (CSM) is an exploratory project aiming to provide “an AppStore for Services.” It is an intelligent online marketplace that facilitates service discovery and acquisition for enterprise customers. Traditional service discovery and acquisition are time-consuming. In the era of OneClick Checkout and pay-as-you-go service plans, users expect services to be purchased online efficiently and conveniently. However, as services are complex and different from software apps, the currently prevailing App Store based on keyword search is inadequate for services. In CSM, exploring and configuring services are an iterative process. Customers provide their requirements in natural language and interact with the system through questioning and answering. Learning from the input, the system can incrementally clarify users’ intention, narrow down the candidate services, and profile the configuration information for the candidates at the same time. CSM’s back end is built around the Services Knowledge Graph (SKG) and leverages data mining technologies to enable the semantic understanding of customers’ requirements. To quantitatively assess the value of CSM, empirical evaluation on real and synthetic datasets and case studies are given to demonstrate the efficacy and effectiveness of the proposed system.

Details DOI

IJCAI Conference 2016 Conference Paper

Probabilistic Rank-One Matrix Analysis with Concurrent Regularization

  • Yang Zhou
  • Haiping Lu

As a classical subspace learning method, Probabilistic PCA (PPCA) has been extended to several bilinear variants for dealing with matrix observations. However, they are all based on the Tucker model, leading to a restricted subspace representation and the problem of rotational ambiguity. To address these problems, this paper proposes a bilinear PPCA method named as Probabilistic Rank-One Matrix Analysis (PROMA). PROMA is based on the CP model, which leads to a more flexible subspace representation and does not suffer from rotational ambiguity. For better generalization, concurrent regularization is introduced to regularize the whole matrix subspace, rather than column and row factors separately. Experiments on both synthetic and real-world data demonstrate the superiority of PROMA in subspace estimation and classification as well as the effectiveness of concurrent regularization in regularizing bilinear PPCAs.

PDF Details

NeurIPS Conference 2009 Conference Paper

Regularized Distance Metric Learning:Theory and Algorithm

  • Rong Jin
  • Shijun Wang
  • Yang Zhou

In this paper, we examine the generalization error of regularized distance metric learning. We show that with appropriate constraints, the generalization error of regularized distance metric learning could be independent from the dimensionality, making it suitable for handling high dimensional data. In addition, we present an efficient online learning algorithm for regularized distance metric learning. Our empirical studies with data classification and face recognition show that the proposed algorithm is (i) effective for distance metric learning when compared to the state-of-the-art methods, and (ii) efficient and robust for high dimensional data.

PDF Details

AAAI Conference 2008 Conference Paper

Using Knowledge Driven Matrix Factorization to Reconstruct Modular Gene Regulatory Network

  • Yang Zhou
  • Xuerui Yang
  • Shireesh Srivastava

Reconstructing gene networks from micro-array data can provide information on the mechanisms that govern cellular processes. Numerous studies have been devoted to addressing this problem. A popular method is to view the gene network as a Bayesian inference network, and to apply structure learning methods to determine the topology of the gene network. There are, however, several shortcomings with the Bayesian structure learning approach for reconstructing gene networks. They include high computational cost associated with analyzing a large number of genes and inefficiency in exploiting prior knowledge of co-regulation that could be derived from Gene Ontology (GO) information. In this paper, we present a knowledge driven matrix factorization (KMF) framework for reconstructing modular gene networks that addresses these shortcomings. In KMF, gene expression data is initially used to estimate the correlation matrix. The gene modules and the interactions among the modules are derived by factorizing the correlation matrix. The prior knowledge in GO is integrated into matrix factorization to help identify the gene modules. An alternating optimization algorithm is presented to efficiently find the solution. Experiments show that our algorithm performs significantly better in identifying gene modules than several state-of-the-art algorithms, and the interactions among the modules uncovered by our algorithm are proved to be biologically meaningful.

PDF Details
v2026.09.13