Author name cluster

Qiao Xiao

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

7 papers

2 author rows

ICLR Conference 2025 Conference Paper

Dynamic Sparse Training versus Dense Training: The Unexpected Winner in Image Corruption Robustness

Boqian Wu
Qiao Xiao
Shunxin Wang
Nicola Strisciuglio
Mykola Pechenizkiy
Maurice van Keulen
Decebal Constantin Mocanu
Elena Mocanu

It is generally perceived that Dynamic Sparse Training opens the door to a new era of scalability and efficiency for artificial neural networks at, perhaps, some costs in accuracy performance for the classification task. At the same time, Dense Training is widely accepted as being the "de facto" approach to train artificial neural networks if one would like to maximize their robustness against image corruption. In this paper, we question this general practice. Consequently, \textit{we claim that}, contrary to what is commonly thought, the Dynamic Sparse Training methods can consistently outperform Dense Training in terms of robustness accuracy, particularly if the efficiency aspect is not considered as a main objective (i.e., sparsity levels between 10\% and up to 50\%), without adding (or even reducing) resource cost. We validate our claim on two types of data, images and videos, using several traditional and modern deep learning architectures for computer vision and three widely studied Dynamic Sparse Training algorithms. Our findings reveal a new yet-unknown benefit of Dynamic Sparse Training and open new possibilities in improving deep learning robustness beyond the current state of the art.

Details

EAAI Journal 2025 Journal Article

Few-shot machine reading comprehension for bridge inspection via domain-specific and task-aware pre-tuning approach

Ren Li
Luyi Zhang
Qiao Xiao
Jianxi Yang
Yu Chen
Shixin Jiang
Di Wang

With the wide application of information technologies in the field of bridge engineering, many electronic bridge inspection reports have been generated. However, due to insufficient research on machine reading comprehension (MRC) in this field, a lot of bridge inspection information, e. g. , structural basic data, inspected defects, and maintenance suggestions, has not been fully used. Especially, it is time-consuming and labor-intensive to pre-train a domain-specific language model from scratch or annotate large-scale question answering corpora, which also brings challenges to the MRC research in this field. To tackle the problems, this paper proposes a novel few-shot MRC approach for bridge inspection based on the idea of data augmentation. The proposed model uses a pre-trained model as backbone, along with introducing a pre-tuning stage to bridge the gaps between general-purpose pre-training and domain-specific MRC tasks. In order to reduce the workload of manual annotation, we present a novel pre-tuning data generation algorithm which is based on the domain-specific question classification and answer prediction neural models. After pre-tuning and fine-tuning, the proposed model achieves efficient bridge inspection MRC. The experimental results show that the proposed model outperforms the mainstream fine-tuning-based approaches and few-shot MRC baseline models in various settings. With 1024 fine-tuning samples, the F1 value and Exact Match (EM) value are 86. 42%, 74. 65%, respectively. Our research work can serve as a foundation for the construction of automatic question answering systems for intelligent bridge management and maintenance.

Details DOI

NeurIPS Conference 2024 Conference Paper

E2ENet: Dynamic Sparse Feature Fusion for Accurate and Efficient 3D Medical Image Segmentation

Boqian Wu
Qiao Xiao
Shiwei Liu
Lu Yin
Mykola Pechenizkiy
Decebal C. Mocanu
Maurice van Keulen
Elena Mocanu

Deep neural networks have evolved as the leading approach in 3D medical image segmentation due to their outstanding performance. However, the ever-increasing model size and computational cost of deep neural networks have become the primary barriers to deploying them on real-world, resource-limited hardware. To achieve both segmentation accuracy and efficiency, we propose a 3D medical image segmentation model called Efficient to Efficient Network (E2ENet), which incorporates two parametrically and computationally efficient designs. i. Dynamic sparse feature fusion (DSFF) mechanism: it adaptively learns to fuse informative multi-scale features while reducing redundancy. ii. Restricted depth-shift in 3D convolution: it leverages the 3D spatial information while keeping the model and computational complexity as 2D-based methods. We conduct extensive experiments on AMOS, Brain Tumor Segmentation and BTCV Challenge, demonstrating that E2ENet consistently achieves a superior trade-off between accuracy and efficiency than prior arts across various resource constraints. %In particular, with a single model and single scale, E2ENet achieves comparable accuracy on the large-scale challenge AMOS-CT, while saving over 69% parameter count and 27% FLOPs in the inference phase, compared with the previousbest-performing method. Our code has been made available at: https: //github. com/boqian333/E2ENet-Medical.

PDF Details DOI

ICLR Conference 2023 Conference Paper

More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity

Shiwei Liu 0003
Tianlong Chen 0001
Xiaohan Chen 0001
Xuxi Chen
Qiao Xiao
Boqian Wu
Tommi Kärkkäinen
Mykola Pechenizkiy

Transformers have quickly shined in the computer vision world since the emergence of Vision Transformers (ViTs). The dominant role of convolutional neural networks (CNNs) seems to be challenged by increasingly effective transformer-based models. Very recently, a couple of advanced convolutional models strike back with large kernels motivated by the local-window attention mechanism, showing appealing performance and efficiency. While one of them, i.e. RepLKNet, impressively manages to scale the kernel size to 31x31 with improved performance, the performance starts to saturate as the kernel size continues growing, compared to the scaling trend of advanced ViTs such as Swin Transformer. In this paper, we explore the possibility of training extreme convolutions larger than 31x31 and test whether the performance gap can be eliminated by strategically enlarging convolutions. This study ends up with a recipe for applying extremely large kernels from the perspective of sparsity, which can smoothly scale up kernels to 61x61 with better performance. Built on this recipe, we propose Sparse Large Kernel Network (SLaK), a pure CNN architecture equipped with sparse factorized 51x51 kernels that can perform on par with or better than state-of-the-art hierarchical Transformers and modern ConvNet architectures like ConvNeXt and RepLKNet, on ImageNet classification as well as a wide range of downstream tasks including semantic segmentation on ADE20K, object detection on PASCAL VOC 2007, and object detection/segmentation on MS COCO. Codes are available at https://github.com/VITA-Group/SLaK.

Details

NeurIPS Conference 2022 Conference Paper

Dynamic Sparse Network for Time Series Classification: Learning What to “See”

Qiao Xiao
Boqian Wu
Yu Zhang
Shiwei Liu
Mykola Pechenizkiy
Elena Mocanu
Decebal Constantin Mocanu

The receptive field (RF), which determines the region of time series to be “seen” and used, is critical to improve the performance for time series classification (TSC). However, the variation of signal scales across and within time series data, makes it challenging to decide on proper RF sizes for TSC. In this paper, we propose a dynamic sparse network (DSN) with sparse connections for TSC, which can learn to cover various RF without cumbersome hyper-parameters tuning. The kernels in each sparse layer are sparse and can be explored under the constraint regions by dynamic sparse training, which makes it possible to reduce the resource cost. The experimental results show that the proposed DSN model can achieve state-of-art performance on both univariate and multivariate TSC datasets with less than 50% computational cost compared with recent baseline methods, opening the path towards more accurate resource-aware methods for time series analyses. Our code is publicly available at: https: //github. com/QiaoXiao7282/DSN.

PDF Details

AAAI Conference 2021 Conference Paper

Distant Transfer Learning via Deep Random Walk

Qiao Xiao
Yu Zhang

Transfer learning, which is to improve the learning performance in the target domain by leveraging useful knowledge from the source domain, often requires that those two domains are very close, which limits its application scope. Recently, distant transfer learning has been studied to transfer knowledge between two distant or even totally unrelated domains via unlabeled auxiliary domains that act as a bridge in the spirit of human transitive inference that two completely unrelated concepts can be connected through gradual knowledge transfer. In this paper, we study distant transfer learning by proposing a DeEp Random Walk basEd distaNt Transfer (DERWENT) method. Different from existing distant transfer learning models that implicitly identify the path of knowledge transfer between the source and target instances through auxiliary instances, the proposed DERWENT model can explicitly learn such paths via the deep random walk technique. Specifically, based on sequences identified by the random walk technique on a data graph where source and target data have no direct connection, the proposed DERWENT model enforces adjacent data points in a sequence to be similar, makes the ending data point be represented by other data points in the same sequence, and considers weighted classification losses of source data. Empirical studies on several benchmark datasets demonstrate that the proposed DERWENT algorithm yields the state-ofthe-art performance.

PDF Details

NeurIPS Conference 2021 Conference Paper

Multi-Objective Meta Learning

Feiyang Ye
Baijiong Lin
Zhixiong Yue
Pengxin Guo
Qiao Xiao
Yu Zhang

Meta learning with multiple objectives has been attracted much attention recently since many applications need to consider multiple factors when designing learning models. Existing gradient-based works on meta learning with multiple objectives mainly combine multiple objectives into a single objective in a weighted sum manner. This simple strategy usually works but it requires to tune the weights associated with all the objectives, which could be time consuming. Different from those works, in this paper, we propose a gradient-based Multi-Objective Meta Learning (MOML) framework without manually tuning weights. Specifically, MOML formulates the objective function of meta learning with multiple objectives as a Multi-Objective Bi-Level optimization Problem (MOBLP) where the upper-level subproblem is to solve several possibly conflicting objectives for the meta learner. To solve the MOBLP, we devise the first gradient-based optimization algorithm by alternatively solving the lower-level and upper-level subproblems via the gradient descent method and the gradient-based multi-objective optimization method, respectively. Theoretically, we prove the convergence properties of the proposed gradient-based optimization algorithm. Empirically, we show the effectiveness of the proposed MOML framework in several meta learning problems, including few-shot learning, domain adaptation, multi-task learning, and neural architecture search. The source code of MOML is available at https: //github. com/Baijiong-Lin/MOML.

PDF Details