Arrow Research search

Author name cluster

Tao Xiang

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

29 papers
1 author row

Possible papers

29

AAAI Conference 2026 Conference Paper

ShadeEdit: A Utility-Preserving and Defense-Evasive Knowledge Manipulation Attack in Federated LLMs

  • Xu Zhang
  • Hangcheng Liu
  • Shangwei Guo
  • Shudong Zhang
  • Tianwei Zhang
  • Tao Xiang

Recent studies reveal that adversaries can manipulate the internal knowledge of large language models (LLMs) on selected topics through model editing, causing attacker-specified harmful or biased outputs when queried about the edited content. Once such tampered LLMs are distributed, they can mislead users on the targeted topics, thereby potentially propagating misinformation or reinforcing stereotypes. However, existing knowledge manipulation attacks rely on the ability to redistribute compromised models, which is infeasible in constrained settings like Federated Instruction Tuning (FedIT), where a central server controls LLM's training and distribution. In this work, we introduce ShadeEdit, the first attack framework that leverages strengthened model editing to enable knowledge manipulation in FedIT scenarios. ShadeEdit introduces two key components to address two challenges posed by the training process of FedIT: (1) a paraphrase-based editing dataset selection strategy to mitigate the dilution from benign updates on malicious ones by constructing a high-quality editing dataset, and (2) an adaptive manipulation mechanism to evade aggregation-based defenses via an adaptive clipping strategy. ShadeEdit achieves an average 99.5% attack success rate over eight robust aggregation algorithms while preserving instruction-following accuracy, demonstrating its strong attack effectiveness and model-utility preservation.

PDF Details DOI

IJCAI Conference 2025 Conference Paper

Advancing Embodied Agent Security: From Safety Benchmarks to Input Moderation

  • Ning Wang
  • Zihan Yan
  • Weiyang Li
  • Chuan Ma
  • He Chen
  • Tao Xiang

Embodied agents exhibit immense potential across a multitude of domains, making the assurance of their behavioral safety a fundamental prerequisite for their widespread deployment. However, existing research predominantly concentrates on the security of general large language models, lacking specialized methodologies for establishing safety benchmarks and input moderation tailored to embodied agents. To bridge this gap, this paper introduces a novel input moderation framework, meticulously designed to safeguard embodied agents. This framework encompasses the entire pipeline, including taxonomy definition, dataset curation, moderator architecture, model training, and rigorous evaluation. Notably, we introduce EAsafetyBench, a meticulously crafted safety benchmark engineered to facilitate both the training and stringent assessment of moderators specifically designed for embodied agents. Furthermore, we propose Pinpoint, an innovative prompt-decoupled input moderation scheme that harnesses a masked attention mechanism to effectively isolate and mitigate the influence of functional prompts on moderation tasks. Extensive experiments conducted on diverse benchmark datasets and models validate the feasibility and efficacy of the proposed approach. The results demonstrate that our methodologies achieve an impressive average detection accuracy of 94. 58%, surpassing the performance of existing state-of-the-art techniques, alongside an exceptional moderation processing time of merely 0. 002 seconds per instance. The source code and datasets can be found at https: //github. com/ZihanYan-CQU/EAsafetyBench.

PDF Details DOI

TMLR Journal 2025 Journal Article

Faster Diffusion Through Temporal Attention Decomposition

  • Haozhe Liu
  • Wentian Zhang
  • Jinheng Xie
  • Francesco Faccio
  • Mengmeng Xu
  • Tao Xiang
  • Mike Zheng Shou
  • Juan-Manuel Perez-Rua

We explore the role of the attention mechanism during inference in text-conditional diffusion models. Empirical observations suggest that cross-attention outputs converge to a fixed point after several inference steps. The convergence time naturally divides the entire inference process into two phases: an initial phase for planning text-oriented visual semantics, which are then translated into images in a subsequent fidelity-improving phase. Cross-attention is essential in the initial phase but almost irrelevant thereafter. Self-attention, however, initially plays a minor role but becomes increasingly important in the second phase. These findings yield a simple and training-free method called TGATE which efficiently generates images by caching and reusing attention outputs at scheduled time steps. Experiments show TGATE’s broad applicability to various existing text-conditional diffusion models which it speeds up by 10-50%. The code of TGATE is available at https://github.com/HaozheLiu-ST/T-GATE.

PDF Details

TMLR Journal 2025 Journal Article

MarDini: Masked Auto-regressive Diffusion for Video Generation at Scale

  • Haozhe Liu
  • Shikun Liu
  • Zijian Zhou
  • Mengmeng Xu
  • Yanping Xie
  • Xiao Han
  • Juan Camilo Perez
  • Ding Liu

We introduce MarDini, a new family of video diffusion models that integrate the advantages of masked auto-regression (MAR) into a unified diffusion model (DM) framework. Here, MAR handles temporal planning, while DM focuses on spatial generation in an asymmetric network design: i) a MAR-based planning model containing most of the parameters generates planning signals for each masked frame using low-resolution input; ii) a lightweight generation model uses these signals to produce high-resolution frames via diffusion de-noising. MarDini’s MAR enables video generation conditioned on any number of masked frames at any frame positions: a single model can handle video interpolation (e.g., masking middle frames), image-to-video generation (e.g., masking from the second frame onward), and video expansion (e.g., masking half the frames). The efficient design allocates most of the computational resources to the low-resolution planning model, making computationally expensive but important spatio-temporal attention feasible at scale. MarDini sets a new state-of-the-art for video interpolation; meanwhile, within few inference steps, it efficiently generates videos on par with those of much more expensive advanced image-to-video models.

PDF Details

NeurIPS Conference 2025 Conference Paper

Transstratal Adversarial Attack: Compromising Multi-Layered Defenses in Text-to-Image Models

  • Chunlong Xie
  • Kangjie Chen
  • Shangwei Guo
  • Shudong Zhang
  • Tianwei Zhang
  • Tao Xiang

Modern Text-to-Image (T2I) models deploy multi-layered defenses to block Not-Safe-For-Work (NSFW) content generation. These defenses typically include sequential layers such as prompt filters, concept erasers and image filters. While existing adversarial attacks have demonstrated vulnerabilities in isolated defense layers, they prove largely ineffective against multi-layered defenses deployed in real-world T2I systems. In this paper, we demonstrate that exploiting overlapping vulnerabilities across these distinct defense layers enables adversaries to systematically bypass the entire safeguard of T2I systems. We propose Transstratal Adversarial Attack (TAA), a novel black-box framework to compromise T2I models with multi-layered protection. It generates transstratal adversarial prompts to evade all defense layers simultaneously. This is accomplished through transstratal adversarial candidate generation using LLMs to fulfill implicit and subjective adversarial requirements against different defense layers, combined with adversarial genetic optimization for efficient black-box search to maximize the bypass rates and generated image harmfulness. Evaluated across 14 T2I models (e. g. , Stable Diffusion, DALL·E, and Midjourney) and 17 safety modules, our attack achieves an average attack success rate of 85. 6\%, surpassing state-of-the-art methods by 73. 5\%. Our findings challenge the isolated design of safety mechanisms and establish the first benchmark for holistic robustness evaluation in multi-layered safeguarded T2I models. The code can be found in https: //github. com/Bluedask/TAA-T2I.

PDF Details

TIST Journal 2024 Journal Article

An Unbiased Risk Estimator for Partial Label Learning with Augmented Classes

  • Jiayu Hu
  • Senlin Shu
  • Beibei Li
  • Tao Xiang
  • Zhongshi He

Partial Label Learning (PLL) is a typical weakly supervised learning task, which assumes each training instance is annotated with a set of candidate labels containing the ground-truth label. Recent PLL methods adopt identification-based disambiguation to alleviate the influence of false positive labels and achieve promising performance. However, they require all classes in the test set to have appeared in the training set, ignoring the fact that new classes will keep emerging in real applications. To address this issue, in this article, we focus on the problem of Partial Label Learning with Augmented Class (PLLAC), where one or more augmented classes are not visible in the training stage but appear in the inference stage. Specifically, we propose an unbiased risk estimator with theoretical guarantees for PLLAC, which estimates the distribution of augmented classes by differentiating the distribution of known classes from unlabeled data and can be equipped with arbitrary PLL loss functions. Besides, we provide a theoretical analysis of the estimation error bound of the estimator, which guarantees the convergence of the empirical risk minimizer to the true risk minimizer as the number of training data tends to infinity. Furthermore, we add a risk-penalty regularization term in the optimization objective to alleviate the influence of the over-fitting issue caused by negative empirical risk. Extensive experiments on benchmark, UCI, and real-world datasets demonstrate the effectiveness of the proposed approach.

Details DOI

NeurIPS Conference 2024 Conference Paper

Beware of Road Markings: A New Adversarial Patch Attack to Monocular Depth Estimation

  • Hangcheng Liu
  • Zhenhu Wu
  • Hao Wang
  • Xingshuo Han
  • Shangwei Guo
  • Tao Xiang
  • Tianwei Zhang

Monocular Depth Estimation (MDE) enables the prediction of scene depths from a single RGB image, having been widely integrated into production-grade autonomous driving systems, e. g. , Tesla Autopilot. Current adversarial attacks to MDE models focus on attaching an optimized adversarial patch to a designated obstacle. Although effective, this approach presents two inherent limitations: its reliance on specific obstacles and its limited malicious impact. In contrast, we propose a pioneering attack to MDE models that \textit{decouples obstacles from patches physically and deploys optimized patches on roads}, thereby extending the attack scope to arbitrary traffic participants. This approach is inspired by our groundbreaking discovery: \textit{various MDE models with different architectures, trained for autonomous driving, heavily rely on road regions} when predicting depths for different obstacles. Based on this discovery, we design the Adversarial Road Marking (AdvRM) attack, which camouflages patches as ordinary road markings and deploys them on roads, thereby posing a continuous threat within the environment. Experimental results from both dataset simulations and real-world scenarios demonstrate that AdvRM is effective, stealthy, and robust against various MDE models, achieving about 1. 507 of Mean Relative Shift Ratio (MRSR) over 8 MDE models. The code is available at \url{https: //github. com/a-c-a-c/AdvRM. git}

PDF Details DOI

NeurIPS Conference 2024 Conference Paper

DI-MaskDINO: A Joint Object Detection and Instance Segmentation Model

  • Zhixiong Nan
  • Xianghong Li
  • Tao Xiang
  • Jifeng Dai

This paper is motivated by an interesting phenomenon: the performance of object detection lags behind that of instance segmentation (i. e. , performance imbalance) when investigating the intermediate results from the beginning transformer decoder layer of MaskDINO (i. e. , the SOTA model for joint detection and segmentation). This phenomenon inspires us to think about a question: will the performance imbalance at the beginning layer of transformer decoder constrain the upper bound of the final performance? With this question in mind, we further conduct qualitative and quantitative pre-experiments, which validate the negative impact of detection-segmentation imbalance issue on the model performance. To address this issue, this paper proposes DI-MaskDINO model, the core idea of which is to improve the final performance by alleviating the detection-segmentation imbalance. DI-MaskDINO is implemented by configuring our proposed De-Imbalance (DI) module and Balance-Aware Tokens Optimization (BATO) module to MaskDINO. DI is responsible for generating balance-aware query, and BATO uses the balance-aware query to guide the optimization of the initial feature tokens. The balance-aware query and optimized feature tokens are respectively taken as the Query and Key&Value of transformer decoder to perform joint object detection and instance segmentation. DI-MaskDINO outperforms existing joint object detection and instance segmentation models on COCO and BDD100K benchmarks, achieving +1. 2 $AP^{box}$ and +0. 9 $AP^{mask}$ improvements compared to SOTA joint detection and segmentation model MaskDINO. In addition, DI-MaskDINO also obtains +1. 0 $AP^{box}$ improvement compared to SOTA object detection model DINO and +3. 0 $AP^{mask}$ improvement compared to SOTA segmentation model Mask2Former.

PDF Details DOI

JBHI Journal 2024 Journal Article

Double Transformer Super-Resolution for Breast Cancer ADC Images

  • Ying Yang
  • Tao Xiang
  • Xiao Lv
  • Lihua Li
  • Lok Ming Lui
  • Tieyong Zeng

Diffusion-weighted imaging (DWI) has been extensively explored in guiding the clinic management of patients with breast cancer. However, due to the limited resolution, accurately characterizing tumors using DWI and the corresponding apparent diffusion coefficient (ADC) is still a challenging problem. In this paper, we aim to address the issue of super-resolution (SR) of ADC images and evaluate the clinical utility of SR-ADC images through radiomics analysis. To this end, we propose a novel double transformer-based network (DTformer) to enhance the resolution of ADC images. More specifically, we propose a symmetric U-shaped encoder-decoder network with two different types of transformer blocks, named as UTNet, to extract deep features for super-resolution. The basic backbone of UTNet is composed of a locally-enhanced Swin transformer block (LeSwin-T) and a convolutional transformer block (Conv-T), which are responsible for capturing long-range dependencies and local spatial information, respectively. Additionally, we introduce a residual upsampling network (RUpNet) to expand image resolution by leveraging initial residual information from the original low-resolution (LR) images. Extensive experiments show that DTformer achieves superior SR performance. Moreover, radiomics analysis reveals that improving the resolution of ADC images is beneficial for tumor characteristic prediction, such as histological grade and human epidermal growth factor receptor 2 (HER2) status.

Details DOI

TIST Journal 2024 Journal Article

Multiple-Instance Learning from Pairwise Comparison Bags

  • Shengjie Zhou
  • Senlin Shu
  • Haobo Wang
  • Hongxin Wei
  • Tao Xiang
  • Beibei Li

Multiple-instance learning (MIL) is a significant weakly supervised learning problem, where the training data consist of bags containing multiple instances and bag-level labels. Most previous MIL research required fully labeled bags. However, collecting such data is challenging due to the labeling costs or privacy concerns. Fortunately, we can easily collect pairwise comparison information, indicating one bag is more likely to be positive than the other. Therefore, we investigate a novel MIL problem about learning a bag-level binary classifier only from pairwise comparison bags. To solve this problem, we display the data generation process and provide a baseline method to train an instance-level classifier based on unlabeled-unlabeled learning. To achieve better performance, we propose a convex formulation to train a bag-level classifier and give a generalization error bound. Comprehensive experiments show that both the baseline method and the convex formulation achieve satisfactory performance, while the convex formulation performs better. 1

Details DOI

NeurIPS Conference 2024 Conference Paper

On-Road Object Importance Estimation: A New Dataset and A Model with Multi-Fold Top-Down Guidance

  • Zhixiong Nan
  • Yilong Chen
  • Tianfei Zhou
  • Tao Xiang

This paper addresses the problem of on-road object importance estimation, which utilizes video sequences captured from the driver's perspective as the input. Although this problem is significant for safer and smarter driving systems, the exploration of this problem remains limited. On one hand, publicly-available large-scale datasets are scarce in the community. To address this dilemma, this paper contributes a new large-scale dataset named Traffic Object Importance (TOI). On the other hand, existing methods often only consider either bottom-up feature or single-fold guidance, leading to limitations in handling highly dynamic and diverse traffic scenarios. Different from existing methods, this paper proposes a model that integrates multi-fold top-down guidance with the bottom-up feature. Specifically, three kinds of top-down guidance factors (i. e. , driver intention, semantic context, and traffic rule) are integrated into our model. These factors are important for object importance estimation, but none of the existing methods simultaneously consider them. To our knowledge, this paper proposes the first on-road object importance estimation model that fuses multi-fold top-down guidance factors with bottom-up feature. Extensive experiments demonstrate that our model outperforms state-of-the-art methods by large margins, achieving 23. 1% Average Precision (AP) improvement compared with the recently proposed model (i. e. , Goal).

PDF Details DOI

AAAI Conference 2024 Conference Paper

United We Stand: Accelerating Privacy-Preserving Neural Inference by Conjunctive Optimization with Interleaved Nexus

  • Qiao Zhang
  • Tao Xiang
  • Chunsheng Xin
  • Hongyi Wu

Privacy-preserving Machine Learning as a Service (MLaaS) enables the powerful cloud server to run its well-trained neural model upon the input from resource-limited client, with both of server's model parameters and client's input data protected. While computation efficiency is critical for the practical implementation of privacy-preserving MLaaS and it is inspiring to witness recent advances towards efficiency improvement, there still exists a significant performance gap to real-world applications. In general, state-of-the-art frameworks perform function-wise efficiency optimization based on specific cryptographic primitives. Although it is logical, such independent optimization for each function makes noticeable amount of expensive operations unremovable and misses the opportunity to further accelerate the performance by jointly considering privacy-preserving computation among adjacent functions. As such, we propose COIN: Conjunctive Optimization with Interleaved Nexus, which remodels mainstream computation for each function to conjunctive counterpart for composite function, with a series of united optimization strategies. Specifically, COIN jointly computes a pair of consecutive nonlinear-linear functions in the neural model by reconstructing the intermediates throughout the whole procedure, which not only eliminates the most expensive crypto operations without invoking extra encryption enabler, but also makes the online crypto complexity independent of filter size. Experimentally, COIN demonstrates 11.2x to 29.6x speedup over various function dimensions from modern networks, and 6.4x to 12x speedup on the total computation time when applied in networks with model input from small-scale CIFAR10 to large-scale ImageNet.

PDF Details DOI

AAAI Conference 2023 Conference Paper

A Generalized Unbiased Risk Estimator for Learning with Augmented Classes

  • Senlin Shu
  • Shuo He
  • Haobo Wang
  • Hongxin Wei
  • Tao Xiang
  • Lei Feng

In contrast to the standard learning paradigm where all classes can be observed in training data, learning with augmented classes (LAC) tackles the problem where augmented classes unobserved in the training data may emerge in the test phase. Previous research showed that given unlabeled data, an unbiased risk estimator (URE) can be derived, which can be minimized for LAC with theoretical guarantees. However, this URE is only restricted to the specific type of one-versus-rest loss functions for multi-class classification, making it not flexible enough when the loss needs to be changed with the dataset in practice. In this paper, we propose a generalized URE that can be equipped with arbitrary loss functions while maintaining the theoretical guarantees, given unlabeled data for LAC. To alleviate the issue of negative empirical risk commonly encountered by previous studies, we further propose a novel risk-penalty regularization term. Experiments demonstrate the effectiveness of our proposed method.

PDF Details DOI

NeurIPS Conference 2023 Conference Paper

HeadSculpt: Crafting 3D Head Avatars with Text

  • Xiao Han
  • Yukang Cao
  • Kai Han
  • Xiatian Zhu
  • Jiankang Deng
  • Yi-Zhe Song
  • Tao Xiang
  • Kwan-Yee K. Wong

Recently, text-guided 3D generative methods have made remarkable advancements in producing high-quality textures and geometry, capitalizing on the proliferation of large vision-language and image diffusion models. However, existing methods still struggle to create high-fidelity 3D head avatars in two aspects: (1) They rely mostly on a pre-trained text-to-image diffusion model whilst missing the necessary 3D awareness and head priors. This makes them prone to inconsistency and geometric distortions in the generated avatars. (2) They fall short in fine-grained editing. This is primarily due to the inherited limitations from the pre-trained 2D image diffusion models, which become more pronounced when it comes to 3D head avatars. In this work, we address these challenges by introducing a versatile coarse-to-fine pipeline dubbed HeadSculpt for crafting (i. e. , generating and editing) 3D head avatars from textual prompts. Specifically, we first equip the diffusion model with 3D awareness by leveraging landmark-based control and a learned textual embedding representing the back view appearance of heads, enabling 3D-consistent head avatar generations. We further propose a novel identity-aware editing score distillation strategy to optimize a textured mesh with a high-resolution differentiable rendering technique. This enables identity preservation while following the editing instruction. We showcase HeadSculpt's superior fidelity and editing capabilities through comprehensive experiments and comparisons with existing methods.

PDF Details

TIST Journal 2023 Journal Article

Towards Query-Efficient Black-Box Attacks: A Universal Dual Transferability-Based Framework

  • Tao Xiang
  • Hangcheng Liu
  • Shangwei Guo
  • Yan Gan
  • Wenjian He
  • Xiaofeng Liao

Adversarial attacks have threatened the application of deep neural networks in security-sensitive scenarios. Most existing black-box attacks fool the target model by interacting with it many times and producing global perturbations. However, all pixels are not equally crucial to the target model; thus, indiscriminately treating all pixels will increase query overhead inevitably. In addition, existing black-box attacks take clean samples as start points, which also limits query efficiency. In this article, we propose a novel black-box attack framework, constructed on a strategy of dual transferability (DT), to perturb the discriminative areas of clean examples within limited queries. The first kind of transferability is the transferability of model interpretations. Based on this property, we identify the discriminative areas of clean samples for generating local perturbations. The second is the transferability of adversarial examples, which helps us to produce local pre-perturbations for further improving query efficiency. We achieve the two kinds of transferability through an independent auxiliary model and do not incur extra query overhead. After identifying discriminative areas and generating pre-perturbations, we use the pre-perturbed samples as better start points and further perturb them locally in a black-box manner to search the corresponding adversarial examples. The DT strategy is general; thus, the proposed framework can be applied to different types of black-box attacks. We conduct extensive experiments to show that, under various system settings, our framework can significantly improve the query efficiency of existing black-box attacks and attack success rates.

Details DOI

AAAI Conference 2022 Conference Paper

Hybrid Graph Neural Networks for Few-Shot Learning

  • Tianyuan Yu
  • Sen He
  • Yi-Zhe Song
  • Tao Xiang

Graph neural networks (GNNs) have been used to tackle the few-shot learning (FSL) problem and shown great potentials under the transductive setting. However under the inductive setting, existing GNN based methods are less competitive. This is because they use an instance GNN as a label propagation/classification module, which is jointly meta-learned with a feature embedding network. This design is problematic because the classifier needs to adapt quickly to new tasks while the embedding does not. To overcome this problem, in this paper we propose a novel hybrid GNN (HGNN) model consisting of two GNNs, an instance GNN and a prototype GNN. Instead of label propagation, they act as feature embedding adaptation modules for quick adaptation of the meta-learned feature embedding to new tasks. Importantly they are designed to deal with a fundamental yet often neglected challenge in FSL, that is, with only a handful of shots per class, any few-shot classifier would be sensitive to badly sampled shots which are either outliers or can cause inter-class distribution overlapping. Extensive experiments show that our HGNN obtains new state-of-the-art on three FSL benchmarks. The code and models are available at https: //github. com/TianyuanYu/HGNN.

PDF Details

AAAI Conference 2021 Conference Paper

A Global Occlusion-Aware Approach to Self-Supervised Monocular Visual Odometry

  • Yao Lu
  • Xiaoli Xu
  • Mingyu Ding
  • Zhiwu Lu
  • Tao Xiang

Self-Supervised monocular visual odometry (VO) is often cast into a view synthesis problem based on depth and camera pose estimation. One of the key challenges is to accurately and robustly estimate depth with occlusions and moving objects in the scene. Existing methods simply detect and mask out regions of occlusions locally by several convolutional layers, and then perform only partial view synthesis in the rest of the image. However, occlusion and moving object detection is an unsolved problem itself which requires global layout information. Inaccurate detection inevitably results in incorrect depth as well as pose estimation. In this work, instead of locally detecting and masking out occlusions and moving objects, we propose to alleviate their negative effects on monocular VO implicitly but more effectively from two global perspectives. First, a multi-scale non-local attention module, consisting of both intra-stage augmented attention and cascaded across-stage attention, is proposed for robust depth estimation given occlusions, alleviating the impacts of occlusions via global attention modeling. Second, adversarial learning is introduced in view synthesis for monocular VO. Unlike existing methods that use pixel-level losses on the quality of synthesized views, we enforce the synthetic view to be indistinguishable from the real one at the scene-level. Such a global constraint again helps cope with occluded and moving regions. Extensive experiments on the KITTI dataset show that our approach achieves new state-of-the-art in both pose estimation and depth recovery.

PDF Details

IJCAI Conference 2021 Conference Paper

Fine-tuning Is Not Enough: A Simple yet Effective Watermark Removal Attack for DNN Models

  • Shangwei Guo
  • Tianwei Zhang
  • Han Qiu
  • Yi Zeng
  • Tao Xiang
  • Yang Liu

Watermarking has become the tendency in protecting the intellectual property of DNN models. Recent works, from the adversary's perspective, attempted to subvert watermarking mechanisms by designing watermark removal attacks. However, these attacks mainly adopted sophisticated fine-tuning techniques, which have certain fatal drawbacks or unrealistic assumptions. In this paper, we propose a novel watermark removal attack from a different perspective. Instead of just fine-tuning the watermarked models, we design a simple yet powerful transformation algorithm by combining imperceptible pattern embedding and spatial-level transformations, which can effectively and blindly destroy the memorization of watermarked models to the watermark samples. We also introduce a lightweight fine-tuning strategy to preserve the model performance. Our solution requires much less resource or knowledge about the watermarking scheme than prior works. Extensive experimental results indicate that our attack can bypass state-of-the-art watermarking solutions with very high success rates. Based on our attack, we propose watermark augmentation techniques to enhance the robustness of existing watermarks.

PDF Details DOI

NeurIPS Conference 2021 Conference Paper

One Loss for All: Deep Hashing with a Single Cosine Similarity based Learning Objective

  • Jiun Tian Hoe
  • Kam Woh Ng
  • Tianyu Zhang
  • Chee Seng Chan
  • Yi-Zhe Song
  • Tao Xiang

A deep hashing model typically has two main learning objectives: to make the learned binary hash codes discriminative and to minimize a quantization error. With further constraints such as bit balance and code orthogonality, it is not uncommon for existing models to employ a large number (>4) of losses. This leads to difficulties in model training and subsequently impedes their effectiveness. In this work, we propose a novel deep hashing model with only $\textit{a single learning objective}$. Specifically, we show that maximizing the cosine similarity between the continuous codes and their corresponding $\textit{binary orthogonal codes}$ can ensure both hash code discriminativeness and quantization error minimization. Further, with this learning objective, code balancing can be achieved by simply using a Batch Normalization (BN) layer and multi-label classification is also straightforward with label smoothing. The result is a one-loss deep hashing model that removes all the hassles of tuning the weights of various losses. Importantly, extensive experiments show that our model is highly effective, outperforming the state-of-the-art multi-loss hashing models on three large-scale instance retrieval benchmarks, often by significant margins.

PDF Details

IJCAI Conference 2021 Conference Paper

Self-Supervised Video Representation Learning with Constrained Spatiotemporal Jigsaw

  • Yuqi Huo
  • Mingyu Ding
  • Haoyu Lu
  • Ziyuan Huang
  • Mingqian Tang
  • Zhiwu Lu
  • Tao Xiang

This paper proposes a novel pretext task for self-supervised video representation learning by exploiting spatiotemporal continuity in videos. It is motivated by the fact that videos are spatiotemporal by nature and a representation learned by detecting spatiotemporal continuity/discontinuity is thus beneficial for downstream video content analysis tasks. A natural choice of such a pretext task is to construct spatiotemporal (3D) jigsaw puzzles and learn to solve them. However, as we demonstrate in the experiments, this task turns out to be intractable. We thus propose Constrained Spatiotemporal Jigsaw (CSJ) whereby the 3D jigsaws are formed in a constrained manner to ensure that large continuous spatiotemporal cuboids exist. This provides sufficient cues for the model to reason about the continuity. Instead of solving them directly, which could still be extremely hard, we carefully design four surrogate tasks that are more solvable. The four tasks aim to learn representations sensitive to spatiotemporal continuity at both the local and global levels. Extensive experiments show that our CSJ achieves state-of-the-art on various benchmarks.

PDF Details DOI

AAAI Conference 2021 Conference Paper

Simple and Effective Stochastic Neural Networks

  • Tianyuan Yu
  • Yongxin Yang
  • Da Li
  • Timothy Hospedales
  • Tao Xiang

Stochastic neural networks (SNNs) are currently topical, with several paradigms being actively investigated including dropout, Bayesian neural networks, variational information bottleneck (VIB) and noise regularized learning. These neural network variants impact several major considerations, including generalization, network compression, robustness against adversarial attack and label noise, and model calibration. However, many existing networks are complicated and expensive to train, and/or only address one or two of these practical considerations. In this paper we propose a simple and effective stochastic neural network (SE-SNN) architecture for discriminative learning by directly modeling activation uncertainty and encouraging high activation variability. Compared to existing SNNs, our SE-SNN is simpler to implement and faster to train, and produces state of the art results on network compression by pruning, adversarial defense, learning with label noise, and model calibration.

PDF Details

NeurIPS Conference 2021 Conference Paper

SOFT: Softmax-free Transformer with Linear Complexity

  • Jiachen Lu
  • Jinghan Yao
  • Junge Zhang
  • Xiatian Zhu
  • Hang Xu
  • Weiguo Gao
  • Chunjing Xu
  • Tao Xiang

Vision transformers (ViTs) have pushed the state-of-the-art for various visual recognition tasks by patch-wise image tokenization followed by self-attention. However, the employment of self-attention modules results in a quadratic complexity in both computation and memory usage. Various attempts on approximating the self-attention computation with linear complexity have been made in Natural Language Processing. However, an in-depth analysis in this work shows that they are either theoretically flawed or empirically ineffective for visual recognition. We further identify that their limitations are rooted in keeping the softmax self-attention during approximations. Specifically, conventional self-attention is computed by normalizing the scaled dot-product between token feature vectors. Keeping this softmax operation challenges any subsequent linearization efforts. Based on this insight, for the first time, a softmax-free transformer or SOFT is proposed. To remove softmax in self-attention, Gaussian kernel function is used to replace the dot-product similarity without further normalization. This enables a full self-attention matrix to be approximated via a low-rank matrix decomposition. The robustness of the approximation is achieved by calculating its Moore-Penrose inverse using a Newton-Raphson method. Extensive experiments on ImageNet show that our SOFT significantly improves the computational efficiency of existing ViT variants. Crucially, with a linear complexity, much longer token sequences are permitted in SOFT, resulting in superior trade-off between accuracy and complexity.

PDF Details

AAAI Conference 2020 Conference Paper

Deep Domain-Adversarial Image Generation for Domain Generalisation

  • Kaiyang Zhou
  • Yongxin Yang
  • Timothy Hospedales
  • Tao Xiang

Machine learning models typically suffer from the domain shift problem when trained on a source dataset and evaluated on a target dataset of different distribution. To overcome this problem, domain generalisation (DG) methods aim to leverage data from multiple source domains so that a trained model can generalise to unseen domains. In this paper, we propose a novel DG approach based on Deep Domain-Adversarial Image Generation (DDAIG). Specifically, DDAIG consists of three components, namely a label classifier, a domain classi- fier and a domain transformation network (DoTNet). The goal for DoTNet is to map the source training data to unseen domains. This is achieved by having a learning objective formulated to ensure that the generated data can be correctly classi- fied by the label classifier while fooling the domain classifier. By augmenting the source training data with the generated unseen domain data, we can make the label classifier more robust to unknown domain changes. Extensive experiments on four DG datasets demonstrate the effectiveness of our approach.

PDF Details

AAAI Conference 2019 Conference Paper

Disjoint Label Space Transfer Learning with Common Factorised Space

  • Xiaobin Chang
  • Yongxin Yang
  • Tao Xiang
  • Timothy M. Hospedales

In this paper, a unified approach is presented to transfer learning that addresses several source and target domain labelspace and annotation assumptions with a single model. It is particularly effective in handling a challenging case, where source and target label-spaces are disjoint, and outperforms alternatives in both unsupervised and semi-supervised settings. The key ingredient is a common representation termed Common Factorised Space. It is shared between source and target domains, and trained with an unsupervised factorisation loss and a graph-based loss. With a wide range of experiments, we demonstrate the flexibility, relevance and efficacy of our method, both in the challenging cases with disjoint label spaces, and in the more conventional cases such as unsupervised domain adaptation, where the source and target domains share the same label-sets.

PDF Details

AAAI Conference 2018 Conference Paper

Deep Reinforcement Learning for Unsupervised Video Summarization With Diversity-Representativeness Reward

  • Kaiyang Zhou
  • Yu Qiao
  • Tao Xiang

Video summarization aims to facilitate large-scale video browsing by producing short, concise summaries that are diverse and representative of original videos. In this paper, we formulate video summarization as a sequential decisionmaking process and develop a deep summarization network (DSN) to summarize videos. DSN predicts for each video frame a probability, which indicates how likely a frame is selected, and then takes actions based on the probability distributions to select frames, forming video summaries. To train our DSN, we propose an end-to-end, reinforcement learningbased framework, where we design a novel reward function that jointly accounts for diversity and representativeness of generated summaries and does not rely on labels or user interactions at all. During training, the reward function judges how diverse and representative the generated summaries are, while DSN strives for earning higher rewards by learning to produce more diverse and more representative summaries. Since labels are not required, our method can be fully unsupervised. Extensive experiments on two benchmark datasets show that our unsupervised method not only outperforms other stateof-the-art unsupervised methods, but also is comparable to or even superior than most of published supervised approaches.

PDF Details

NeurIPS Conference 2018 Conference Paper

Domain-Invariant Projection Learning for Zero-Shot Recognition

  • An Zhao
  • Mingyu Ding
  • Jiechao Guan
  • Zhiwu Lu
  • Tao Xiang
  • Ji-Rong Wen

Zero-shot learning (ZSL) aims to recognize unseen object classes without any training samples, which can be regarded as a form of transfer learning from seen classes to unseen ones. This is made possible by learning a projection between a feature space and a semantic space (e. g. attribute space). Key to ZSL is thus to learn a projection function that is robust against the often large domain gap between the seen and unseen classes. In this paper, we propose a novel ZSL model termed domain-invariant projection learning (DIPL). Our model has two novel components: (1) A domain-invariant feature self-reconstruction task is introduced to the seen/unseen class data, resulting in a simple linear formulation that casts ZSL into a min-min optimization problem. Solving the problem is non-trivial, and a novel iterative algorithm is formulated as the solver, with rigorous theoretic algorithm analysis provided. (2) To further align the two domains via the learned projection, shared semantic structure among seen and unseen classes is explored via forming superclasses in the semantic space. Extensive experiments show that our model outperforms the state-of-the-art alternatives by significant margins.

PDF Details

AAAI Conference 2016 Conference Paper

Robust Complex Behaviour Modeling at 90Hz

  • Xiangyu Kong
  • Yizhou Wang
  • Tao Xiang

Modeling complex crowd behaviour for tasks such as rare event detection has received increasing interest. However, existing methods are limited because (1) they are sensitive to noise often resulting in a large number of false alarms; and (2) they rely on elaborate models leading to high computational cost thus unsuitable for processing a large number of video inputs in real-time. In this paper, we overcome these limitations by introducing a novel complex behaviour modeling framework, which consists of a Binarized Cumulative Directional (BCD) feature as representation, novel spatial and temporal context modeling via an iterative correlation maximization, and a set of behaviour models, each being a simple Bernoulli distribution. Despite its simplicity, our experiments on three benchmark datasets show that it significantly outperforms the state-of-the-art for both temporal video segmentation and rare event detection. Importantly, it is extremely efficient — reaches 90Hz on a normal PC platform using MAT- LAB.

PDF Details

TCS Journal 2014 Journal Article

Period distribution of generalized discrete Arnold cat map

  • Fei Chen
  • Kwok-wo Wong
  • Xiaofeng Liao
  • Tao Xiang

The generalized discrete Arnold cat map is adopted in various cryptographic and steganographic applications where chaos is employed. In this paper, we analyze the period distribution of this map. A systematic approach for addressing the general period distribution problem for any integer value of the modulus N is outlined, followed by a complete analysis for the case of prime N. The analysis is based on similar techniques studying linear feedback shift register (LFSR) sequences. Together with our previous results when N is a power of a prime [1, 2], the period distribution of the cat map is characterized nearly completely for any integer N. Our results are also useful for evaluating the security of the cryptographic and steganographic algorithms based on the cat map as well as computing all unstable periodic orbits of the chaotic Arnold cat map.

Details DOI
v2026.09.13