Gang Li

EAAI Journal 2026 Journal Article

Adaptive neural network control using prescribed performance and event-triggered for path tracking control of autonomous vehicle

• Yongfu Wang
• Sucai Zhang
• Gang Li

This paper investigates the problem of path tracking control for autonomous vehicle in conjunction with steer-by-wire system. Firstly, the dynamic equation of the steer-by-wire system is combined with the dynamic equation of path tracking error. Before the controller design, a state observer is designed to estimate the difficult-to-measure vehicle sideslip angle and steering angular velocity. Meanwhile, the external disturbance and nonlinear friction present in the steer-by-wire system are estimated using a disturbance observer and a radial basis function neural network, respectively. Then, to ensure the steady state and transient performance of the path tracking error within the specified boundaries, a prescribed performance function is constructed with the user-designed tracking accuracy and settling time. Finally, the controller based on the backstepping control is designed and constructed with a dynamic event-triggered mechanism and variable threshold parameters to reduce the transmission frequency of the signal. The Lyapunov stability theory shows that all signals of the system are bounded, and the tracking error converges to a preset range within a finite time. The effectiveness of the proposed control scheme is verified through various simulations, hardware-in-the-loop experiments, and real-time vehicle experiments.

AAAI Conference 2026 Conference Paper

DLDA: Unified Dual-Level Domain Adaptation for Low-Light Object Detection

• Jiayi Hu
• Qian Zhao
• Gang Li

Low-light object detection faces significant challenges due to the substantial domain shift between normal-light and low-light conditions. Prior works often enhance low-light images before detection, but this preprocessing can introduce artifacts that degrade detection performance since it focuses on human visual quality rather than task-specific features. Other methods incorporate illumination-aware modules for low-light feature learning, yet their scalability is limited by the scarcity of annotated low-light datasets. To overcome these limitations, we propose a unified Dual-Level Domain Adaptation (DLDA) framework that jointly addresses image-level and feature-level domain discrepancies. Specifically, we introduce a luminance-aware contrastive translation module that synthesizes target-style low-light images while preserving structural details, enabling effective image-level adaptation. Building on this, we further design a multi-scale conditional adversarial alignment strategy that promotes semantic consistency across feature hierarchies to enhance domain-invariant feature extraction. Extensive experiments on multiple low-light detection benchmarks demonstrate that DLDA achieves state-of-the-art performance, exhibiting strong robustness and generalization.

AAAI Conference 2026 Conference Paper

FedCure: Mitigating Participation Bias in Semi-Asynchronous Federated Learning with Non-IID Data

• Yue Chen
• Jianfeng Lu
• Shuqin Cao
• Wei Wang
• Gang Li
• Guanghui Wen

While semi-asynchronous federated learning (SAFL) combines the efficiency of synchronous training with the flexibility of asynchronous updates, it inherently suffers from participation bias, which is further exacerbated by non-IID data distributions. More importantly, hierarchical architecture shifts participation from individual clients to client groups, thereby further intensifying this issue. Despite notable advancements in SAFL research, most existing works still focus on conventional cloud-end architectures while largely overlooking the critical impact of non-IID data on scheduling across the cloud–edge–client hierarchy. To tackle these challenges, we propose FedCure, an innovative semiasynchronous Federated learning framework that leverages Coalition construction and participation-aware scheduling to mitigate participation bias with non-IID data. Specifically, FedCure operates through three key rules: (1) a preference rule that optimizes coalition formation by maximizing collective benefits and establishing theoretically stable partitions to reduce non-IID-induced performance degradation; (2) a scheduling rule that integrates the virtual queue technique with Bayesian-estimated coalition dynamics, mitigating efficiency loss while ensuring mean rate stability; and (3) a resource allocation rule that enhances computational efficiency by optimizing client CPU frequencies based on estimated coalition dynamics while satisfying delay requirements. Comprehensive experiments on four real-world datasets demonstrate that FedCure improves accuracy by up to 5.1x compared with four state-of-the-art baselines, while significantly enhancing efficiency with the lowest coefficient of variation 0.0223 for per-round latency and maintaining long-term balance across diverse scenarios.

EAAI Journal 2026 Journal Article

Low-light image segmentation using sine-curve enhancement and cross-resolution attention mechanism

• Linkang Xu
• Gang Li
• Xiangxin Ji
• Yue Song

Existing segmentation networks fail to effectively address drone applications in low-light environments, as they struggle to extract reliable semantic and edge information. To address this issue, we propose an end-to-end low-light scene segmentation network. The network employs a novel semantic signal enhancement module, utilizing the characteristics of the sine (SIN) curve function to improve semantic specificity in low-light images, thereby enhancing the feature extraction capability of the backbone network. Additionally, we integrate Hilbert dimensionality reduction to create a cross-resolution attention module that fuses edge information with high level semantic extraction, enhancing edge details. Experimental results indicate that our method yields improvements of 7. 0% and 7. 6% in mean intersection over union (mIoU) and overall accuracy (Acc) over the Low-Light Indoor Scene Understanding (LISU) benchmark on the Low-Light RGB-D (LLRGBD) dataset, achieving 61. 83% and 75. 82%, respectively. The method also performs consistently well on the Full-Time Multi-Modality (FMB) dataset and NightCity, demonstrating strong robustness across diverse low-light scenarios. The code is available at https: //github. com/mumu-k/LLightSegNet.

EAAI Journal 2026 Journal Article

Normality-enhanced knowledge distillation network for unsupervised industrial anomaly detection

• Gang Li
• Tianjiao Chen
• Jin Wan
• Mingle Zhou
• Delong Han
• Min Li

Applications in Engineering: Unsupervised Anomaly Detection (UAD) is essential for industrial surface defect detection since it eliminates the need for extensive and costly labeled data for training while effectively identifying previously unknown defects. Knowledge Distillation (KD)-based UAD has demonstrated effective results, which utilizes the feature differences between the teacher network (T-Net) and the student network (S-Net) to detect anomalies. However, existing methods struggle with normality forgetting and fail to capture subtle features of normal samples during the learning process, which hampers their ability to accurately detect anomalies. To address these issues, we propose a Normality-Enhanced Knowledge Distillation Network (NEKD). The contribution of artificial intelligence: Firstly, we propose a Memory Expert Mechanism (MEM) to strengthen the normality of features in the S-Net by recalling memorized normal feature information. Secondly, the Context-Aware Visual State Space (CA-VSS) block is proposed to enrich the feature representation of normal samples by extracting both global and local features. To enhance the S-Net’s ability to represent normal samples, we present a Dual-domain Consistency-constrained Loss (DCL) that constrains the feature distillation process. Extensive experiments on the MVTec Anomaly Detection (MVTec AD), BeanTech Anomaly Detection (BTAD), and MVTec 3D Anomaly Detection (MVTec 3D-AD), benchmark datasets demonstrate that our proposed method achieves state-of-the-art performance compared to other competitors.

AAAI Conference 2026 Conference Paper

OPTION: An Online Pricing Strategy for Asynchronous Federated Learning Against Free-Riding Attacks

• Bangqi Pan
• Jianfeng Lu
• Shuqin Cao
• Xiao Zhang
• Gang Li
• Guanghui Wen

Asynchronous Federated Learning (AFL) is acclaimed for accelerating collaborative training on heterogeneous systems by eliminating the wait for stragglers. While current solutions focus on improving convergence amidst update delays, they neglect how delayed aggregation fosters free-riding attacks, allowing malicious clients to easily extract the global model without contribution. This behavior results in significant fairness issues and performance degradation. To address this challenge, we propose OPTION, the first online pricing strategy tailored to mitigate free-riding in AFL. OPTION establishes an economic model in which access to model updates is purchased using credits earned from verified contributions. Specifically, OPTION values each model update according to its marginal performance gain and training cost, and subsequently necessitates a download fee from each client based on the Hotelling model to prevent zero-cost acquisition. Moreover, OPTION rewards clients for successful updates under non-arbitrage constraints, effectively balancing individual utility and task budget. To maximize the average model performance while satisfying these conditions, OPTION leverages the Lyapunov drift framework and a probabilistic sampling-based algorithm to optimize the pricing parameters. Extensive experimental results on three real-world datasets demonstrate that OPTION effectively mitigates freeriding attacks in AFL, increases the number of valid updates by at least 23.97%, and achieves a model accuracy improvement of at least 3.01% compared to state-of-the-art baselines.

AAAI Conference 2026 Conference Paper

OursFed: Provable Group Fairness-Aware Federated Learning Against Distrust and Fragility

• Yun Xin
• Jianfeng Lu
• Gang Li
• Shuqin Cao
• Guanghui Wen
• Kehao Wang

With the increasing application of high-stakes decisionmaking application in Federated Learning (FL), ensuring fairness across different populations to prevent biases against certain groups has become crucial. However, achieving group fairness (GF) in FL presents a formidable challenge due to its decentralization, which complicates the global GF estimation by the server. Moreover, distrust and fragility hinder the server from gathering GF values from unreliable clients. This challenge motivates our proposal of OursFed, a provable GF-aware FL framework that integrates a privacy pairbased contract and robust GF estimation method to address issues of distrust and fragility. Methodologically, we categorize client unreliability into two categories: active unreliability stemming from distrust and passive unreliability arising from fragility. To mitigate active unreliability, we design a privacy pair-based contract to guarantee truthful GF reporting, and enhance multivariate analysis by identifying relationships among multiple private data. To counteract passive unreliability, we develop a robust GF estimation using non-parametric techniques to smooth data and estimate probability densities and regression functions, improving per-client GF accuracy under multi-dimensional data perturbation. Theoretically, we demonstrate the efficacy of OursFed by analyzing its convergence, GF stability, and accuracy deviation. Experimentally, evaluations on two real datasets show that OursFed improves GF by 28.61% with at most 2.7% trade-off versus state-ofthe-art baselines, and synthetic experiments further confirm its effectiveness in handling fragility and distrust.

TMLR Journal 2026 Journal Article

Single-loop Algorithms for Stochastic Non-Convex Optimization with Weakly-Convex Constraints

• Ming Yang
• Gang Li
• Quanqi Hu
• Qihang Lin
• Tianbao Yang

Constrained optimization with multiple functional inequality constraints has significant applications in machine learning. This paper examines a crucial subset of such problems where both the objective and constraint functions are weakly convex. Existing methods often face limitations, including slow convergence rates or reliance on double-loop algorithmic designs. To overcome these challenges, we introduce a novel single-loop penalty-based stochastic algorithm. Following the classical exact penalty method, our approach employs a hinge-based penalty, which permits the use of a constant penalty parameter, enabling us to achieve a state-of-the-art complexity for finding an approximate Karush-Kuhn-Tucker (KKT) solution. We further extend our algorithm to address finite-sum coupled compositional objectives, which are prevalent in artificial intelligence applications, establishing improved complexity over existing approaches. Finally, we validate our method through experiments on fair learning with receiver operating characteristic (ROC) fairness constraints and continual learning with non-forgetting constraints.

AAAI Conference 2026 Conference Paper

TOPOGRAPH: Topology-Preserving Graph Reduction with Adaptive Structure for Persistent Homology

• Zonghao Chen
• Yuncheng Jiang
• Gang Li

Topological Data Analysis (TDA) provides artificial intelligence (AI) systems with mathematically rigorous geometric descriptors through Persistent Homology (PH), capturing essential shape characteristics in high-dimensional data. Yet, PH’s combinatorial complexity and sensitivity to outliers hinder its scalability and reliability, especially for Intrinsic PH (IPH) that relies on accurate geodesic distances. While stateof-the-art landmark-based subsampling methods, PH Landmarks, ameliorate computational costs and improve outlier robustness by selecting representative points based on local PH scores, it remain computationally intensive and at low sampling rates struggle to reconstruct the global topology. In this work, we introduce TOPOGRAPH, a simple yet powerful framework that preserves intrinsic topology. The resulting coarsened graph supports efficient IPH computations using Fermat distances. Experiments on both synthetic and realworld datasets show that TOPOGRAPH outperforms stateof-the-art sampling-based methods by achieving an order-ofmagnitude speedup and substantially improved topological fidelity in persistence diagrams, demonstrating its ability for robust and scalable topological data analysis.

AAAI Conference 2025 Conference Paper

Bridging Training and Execution via Dynamic Directed Graph-Based Communication in Cooperative Multi-Agent Systems

• Zhuohui Zhang
• Bin He
• Bin Cheng
• Gang Li

Multi-agent systems must learn to communicate and understand interactions between agents to achieve cooperative goals in partially observed tasks. However, existing approaches lack a dynamic directed communication mechanism and rely on global states, thus diminishing the role of communication in centralized training. Thus, we propose the Transformer-based graph coarsening network (TGCNet), a novel multi-agent reinforcement learning (MARL) algorithm. TGCNet learns the topological structure of a dynamic directed graph to represent the communication policy and integrates graph coarsening networks to approximate the representation of global state during training. It also utilizes the Transformer decoder for feature extraction during execution. Experiments on multiple cooperative MARL benchmarks demonstrate state-of-the-art performance compared to popular MARL algorithms. Further ablation studies validate the effectiveness of our dynamic directed graph communication mechanism and graph coarsening networks.

IJCAI Conference 2025 Conference Paper

DaringFed: A Dynamic Bayesian Persuasion Pricing for Online Federated Learning Under Two-sided Incomplete Information

• Yun Xin
• Jianfeng Lu
• Shuqin Cao
• Gang Li
• Haozhao Wang
• Guanghui Wen

Online Federated Learning (OFL) is a real-time learning paradigm that sequentially executes parameter aggregation immediately for each random arriving client. To motivate clients to participate in OFL, it is crucial to offer appropriate incentives to offset the training resource consumption. However, the design of incentive mechanisms in OFL is constrained by the dynamic variability of Two-sided Incomplete Information (TII) concerning resources, where the server is unaware of the clients’ dynamically changing computational resources, while clients lack knowledge of the real-time communication resources allocated by the server. To incentivize clients to participate in training by offering dynamic rewards to each arriving client, we design a novel Dynamic Bayesian persuasion pricing for online Federated learning (DaringFed) under TII. Specifically, we begin by formulating the interaction between the server and clients as a dynamic signaling and pricing allocation problem within a Bayesian persuasion game, and then demonstrate the existence of a unique Bayesian persuasion Nash equilibrium. By deriving the optimal design of DaringFed under one-sided incomplete information, we further analyze the approximate optimal design of DaringFed with a specific bound under TII. Finally, extensive evaluation conducted on real datasets demonstrate that DaringFed optimizes accuracy and converges speed by 16. 99%, while experiments with synthetic datasets validate the convergence of estimate unknown values and the effectiveness of DaringFed in improving the server’s utility by up to 12. 6%.

NeurIPS Conference 2025 Conference Paper

Denoising Trajectory Biases for Zero-Shot AI-Generated Image Detection

• Yachao Liang
• Min Yu
• Gang Li
• Jianguo Jiang
• Fuqiang Du
• Jingyuan Li
• Lanchi Xie
• Zhen Xu

The rapid advancement of generative models has led to the widespread emergence of highly realistic synthetic images, making the detection of AI-generated content increasingly critical. In particular, diffusion models have recently achieved unprecedented levels of visual fidelity, further raising concerns. While most existing approaches rely on supervised learning, zero-shot detection methods have attracted growing interest due to their ability to bypass data collection and maintenance. Nevertheless, the performance of current zero-shot methods remains limited. In this paper, we introduce a novel zero-shot AI-generated image detection method. Unlike previous works that primarily focus on identifying artifacts in the final generated images, our work explores features within the image generation process that can be leveraged for detection. Specifically, we simulate the image sampling process via diffusion-based inversion and observe that the denoising outputs of generated images converge to the target image more rapidly than those of real images. Inspired by this observation, we compute the similarity between the original image and the outputs along the denoising trajectory, which is then used as an indicator of image authenticity. Since our method requires no training on any generated images, it avoids overfitting to specific generative models or dataset biases. Experiments across a wide range of generators demonstrate that our method achieves significant improvements over state-of-the-art supervised and zero-shot counterparts.

NeurIPS Conference 2025 Conference Paper

DisCO: Reinforcing Large Reasoning Models with Discriminative Constrained Optimization

• Gang Li
• Ming Lin
• Tomer Galanti
• Zhengzhong Tu
• Tianbao Yang

The recent success and openness of DeepSeek-R1 have brought widespread attention to Group Relative Policy Optimization (GRPO) as a reinforcement learning method for large reasoning models (LRMs). In this work, we analyze the GRPO objective under a binary reward setting and reveal an inherent limitation of question-level difficulty bias arising from its group relative advantage function. We also identify a connection between GRPO and traditional discriminative methods in supervised learning. Motivated by these insights, we introduce a new Discriminative Constrained Optimization (DisCO) framework for reinforcing LRMs, grounded in the principle of discriminative learning: increasing the scores of positive answers while decreasing those of negative ones. The main differences between DisCO and GRPO and its recent variants are: (1) it replaces the group relative objective with a discriminative objective defined by a scoring function; (2) it abandons clipping-based surrogates in favor of non-clipping RL surrogate objectives used as scoring functions; (3) it employs a simple yet effective constrained optimization approach to enforce the KL divergence constraint. As a result, DisCO offers notable advantages over GRPO and its variants: (i) it completely eliminates difficulty bias by adopting discriminative objectives; (ii) it addresses the entropy instability in GRPO and its variants through the use of non-clipping scoring functions and a constrained optimization approach, yielding long and stable training dynamics; (iii) it allows the incorporation of advanced discriminative learning techniques to address data imbalance, where a significant number of questions have more negative than positive generated answers during training. Our experiments on enhancing the mathematical reasoning capabilities of SFT-finetuned models show that DisCO significantly outperforms GRPO and its improved variants such as DAPO, achieving average gains of 7\% over GRPO and 6\% over DAPO across six benchmark tasks for a 1. 5B model.

NeurIPS Conference 2025 Conference Paper

Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models

• Yulei Qin
• Gang Li
• Zongyi Li
• Zihan Xu
• Yuchen Shi
• Zhekai Lin
• Xiao Cui
• Ke Li

Existing large language models (LLMs) face challenges of following complex instructions, especially when multiple constraints are present and organized in paralleling, chaining, and branching structures. One intuitive solution, namely chain-of-thought (CoT), is expected to universally improve capabilities of LLMs. However, we find that the vanilla CoT exerts a negative impact on performance due to its superficial reasoning pattern of simply paraphrasing the instructions. It fails to peel back the compositions of constraints for identifying their relationship across hierarchies of types and dimensions. To this end, we propose RAIF, a systematic method to boost LLMs in dealing with complex instructions via incentivizing reasoning for test-time compute scaling. First, we stem from the decomposition of complex instructions under existing taxonomies and propose a reproducible data acquisition method. Second, we exploit reinforcement learning (RL) with verifiable rule-centric reward signals to cultivate reasoning specifically for instruction following. We address the shallow, non-essential nature of reasoning under complex instructions via sample-wise contrast for superior CoT enforcement. We also exploit behavior cloning of experts to facilitate steady distribution shift from fast-thinking LLMs to skillful reasoners. Extensive evaluations on seven comprehensive benchmarks confirm the validity of the proposed method, where a 1. 5B LLM achieves 11. 74% gains with performance comparable to a 8B LLM. Evaluation on OOD constraints also confirms the generalizability of our RAIF.

NeurIPS Conference 2025 Conference Paper

Incentivizing Truthful Language Models via Peer Elicitation Games

• Baiting Chen
• Tong Zhu
• Jiale Han
• Lexin Li
• Gang Li
• Xiaowu Dai

Large Language Models (LLMs) have demonstrated strong generative capabilities but remain prone to inconsistencies and hallucinations. We introduce Peer Elicitation Games (PEG), a training-free, game-theoretic framework for aligning LLMs through a peer elicitation mechanism involving a generator and multiple discriminators instantiated from distinct base models. Discriminators interact in a peer evaluation setting, where utilities are computed using a determinant-based mutual information score that provably incentivizes truthful reporting without requiring ground-truth labels. We establish theoretical guarantees showing that each agent, via online learning, achieves sublinear regret in the sense their cumulative performance approaches that of the best fixed truthful strategy in hindsight. Moreover, we prove last-iterate convergence to a truthful Nash equilibrium, ensuring that the actual policies used by agents converge to stable and truthful behavior over time. Empirical evaluations across multiple benchmarks demonstrate significant improvements in factual accuracy. These results position PEG as a practical approach for eliciting truthful behavior from LLMs without supervision or fine-tuning.

EAAI Journal 2025 Journal Article

MemMambaAD: Memory-augmented state space model for multivariate time series anomaly detection

• Gang Li
• Mingchao Ge
• Jin Wan
• Delong Han
• Min Li
• Mingle Zhou

Multivariate time series anomaly detection focuses on recognizing abnormal patterns to reduce system failures and improve production efficiency and product quality. Accurately detecting anomalies in data remains challenging because existing reconstruction-based methods are prone to overfitting. Recently, reconstruction methods guided by memory modules have been used to address this issue. However, these methods still suffer from insufficient feature extraction of time series prototypes and inadequate storage of normal sample prototype patterns in memory modules. To address these issues, we propose a memory-augmented state space model for multivariate time series anomaly detection. Specifically, we introduce a sequence decomposition state space model-temporal convolutional encoder, which independently extracts trend and seasonal features of multivariate time series in global and local manner, capturing the intrinsic patterns of time series more comprehensively. In addition, we propose a dynamic memory update mechanism, which flexibly updates the memory item through the memory selection mechanism, to more accurately record the prototype pattern of normal samples to improve anomaly detection performance. Extensive experiments on five benchmark datasets demonstrate that our method achieves state-of-the-art performance and reduces memory usage compared with other methods.

TMLR Journal 2025 Journal Article

Multi-Output Distributional Fairness via Post-Processing

• Gang Li
• Qihang Lin
• Ayush Ghosh
• Tianbao Yang

The post-processing approaches are becoming prominent techniques to enhance machine learning models' fairness because of their intuitiveness, low computational cost, and excellent scalability. However, most existing post-processing methods are designed for task-specific fairness measures and are limited to single-output models. In this paper, we introduce a post-processing method for multi-output models, such as the ones used for multi-task/multi-class classification and representation learning, to enhance a model's distributional parity, a task-agnostic fairness measure. Existing methods for achieving distributional parity rely on the (inverse) cumulative density function of a model’s output, restricting their applicability to single-output models. Extending previous works, we propose to employ optimal transport mappings to move a model's outputs across different groups towards their empirical Wasserstein barycenter. An approximation technique is applied to reduce the complexity of computing the exact barycenter and a kernel regression method is proposed to extend this process to out-of-sample data. Our empirical studies evaluate the proposed approach against various baselines on multi-task/multi-class classification and representation learning tasks, demonstrating the effectiveness of the proposed approach.

EAAI Journal 2025 Journal Article

Scene text image super-resolution with semantic-aware interaction

• Mingle Zhou
• Wenlong Liu
• Jin Wan
• Delong Han
• Min Li
• Gang Li

Scene text image super-resolution aims to enhance the resolution of images containing text in various scenes, which amplifies the prominence of the text and improves its recognizability. Existing methods struggle to accurately localize text regions in high-noise environments, which hampers their ability to effectively implement targeted super-resolution. To address these issues, we propose a Text image Super-resolution Semantic-aware Interaction Network (TSSIN) by embedding a text region segmentation network. First, we propose to use a pre-trained text region segmentation network (TRSN) to extract text region information. This approach semantically guides our model to address the challenge of the model not performing targeted super-resolution processing of text in high-noise environments. Secondly, we propose a multi-modal semantic information interaction module (MSIIM) to mitigate the issue of insufficient global information exchange. Comprehensive experiments conducted on the TextZoom dataset demonstrate that our TSSIN significantly enhances image quality. Furthermore, it shows a clear superiority over state-of-the-art methods on TextZoom, achieving an average text recognition accuracy improvement of +1. 0% over Transformer-Based Super-Resolution Network (TBSRN) (49. 6%, 56. 2%, 60. 1% vs. 48. 1%, 55. 9%, 58. 9%), +0. 83% over Parallelly Contextual Attention Network (PCAN), and +1. 46% over Text Prior Guided Super-Resolution (TPGSR), as evaluated by three pre-trained text recognition models. Code is available at https: //github. com/ads2d/TSSIN.

YNIMG Journal 2025 Journal Article

STF: A spherical transformer for versatile cortical surfaces applications

• Jiale Cheng
• Fenqiang Zhao
• Zhengwang Wu
• Xinrui Yuan
• Li Wang
• John H Gilmore
• Weili Lin
• Xin Zhang

Inspired by the remarkable success of attention mechanisms in various applications, there is a growing need to adapt the Transformer architecture from conventional Euclidean domains to non-Euclidean spaces commonly encountered in medical imaging. Structures such as brain cortical surfaces, represented by triangular meshes, exhibit spherical topology and present unique challenges. To address this, we propose the Spherical Transformer (STF), a versatile backbone that leverages self-attention for analyzing cortical surface data. Our approach involves mapping cortical surfaces onto a sphere, dividing them into overlapping patches, and tokenizing both patches and vertices. By performing self-attention at patch and vertex levels, the model simultaneously captures global dependencies and preserves fine-grained contextual information within each patch. Overlapping regions between neighboring patches naturally enable efficient cross-patch information sharing. To handle longitudinal cortical surface data, we introduce the spatiotemporal self-attention mechanism, which jointly captures spatial context and temporal developmental patterns within a single layer. This innovation enhances the representational power of the model, making it well-suited for dynamic surface data. We evaluate the Spherical Transformer on key tasks, including cognition prediction at the surface level and two vertex-level tasks: cortical surface parcellation and cortical property map prediction. Across these applications, our model consistently outperforms state-of-the-art methods, demonstrating its ability to effectively model global dependencies and preserve detailed spatial information. The results highlight its potential as a general-purpose framework for cortical surface analysis.

EAAI Journal 2025 Journal Article

Three-dimensional reconstruction and fracture segmentation based on X-ray and computed tomography paired dataset

• Yuan Gao
• Yuan Zhou
• Da Chen
• Jiachen Li
• Mingle Zhou
• Gang Li
• Yunbo Gu
• Jean-Louis Coatrieux

In some orthopedic surgeries, the use of three-dimensional (3D) computed tomography (CT) scanning technology is not feasible due to scene limitations, leaving doctors to rely on two-dimensional (2D) X-ray images for real-time diagnosis. However, X-ray images lack 3D information, making accurate diagnosis challenging. Developing an algorithm to convert 2D X-ray images into 3D CT images, while simultaneously combining high-quality 3D reconstruction with precise fracture segmentation, offers a promising solution to the problem. In this study, we propose a novel artificial intelligence (AI)-driven framework named 3D reconstruction and segment anything model (3DRecSAM). The reconstruction image enhancer (RIE) is designed to achieve high-precision 3D reconstruction and provide high-quality feature initialization for fracture segmentation. Meanwhile, the mamba segment anything model (MSAM), based on the segment anything model (SAM) architecture, is developed for accurate fracture segmentation. We introduce a Kolmogorov–Arnold network (KAN)-based attention fusion module (KAF), which facilitates the joint optimization of the RIE reconstruction network and the MSAM segmentation network. Furthermore, the selective scanning mamba with KAN (SKM) is incorporated to enhance feature extraction for both RIE and MSAM. Mamba efficiently captures long-range dependencies and sequential patterns, while KAN’s learnable activation functions facilitate adaptive feature fusion and non-linear representation. To train and evaluate 3DRecSAM, we introduce the real X-ray and CT paired dataset (XCPData), which is publicly available on GitHub: https: //github. com/YuanGao1201/XCPData.

TMLR Journal 2025 Journal Article

Unleashing the Power of Data Tsunami: A Comprehensive Survey on Data Assessment and Selection for Instruction Tuning of Language Models

• Yulei Qin
• Yuncheng Yang
• Pengcheng Guo
• Gang Li
• Hang Shao
• Yuchen Shi
• Zihan Xu
• Yun Gu

Instruction tuning plays a critical role in aligning large language models (LLMs) with human preference. Despite the vast amount of open instruction datasets, naively training a LLM on all existing instructions may not be optimal and practical. To pinpoint the most beneficial datapoints, data assessment and selection methods have been proposed in the fields of natural language processing (NLP) and deep learning. However, under the context of instruction tuning, there still exists a gap in knowledge on what kind of data evaluation metrics can be employed and how they can be integrated into the selection mechanism. To bridge this gap, we present a comprehensive review on existing literature of data assessment and selection especially for instruction tuning of LLMs. We systematically categorize all applicable methods into quality-based, diversity-based, and importance-based ones where a unified, fine-grained taxonomy is structured. For each category, representative methods are elaborated to describe the landscape of relevant research. In addition, comparison between the latest methods is conducted on their officially reported results to provide in-depth discussions on their limitations. Finally, we summarize the open challenges and propose the promosing avenues for future studies. All related contents are available at https://github.com/yuleiqin/fantastic-data-engineering.

YNIMG Journal 2024 Journal Article

A joint brain extraction and image quality assessment framework for fetal brain MRI slices

• Wenhao Zhang
• Xin Zhang
• Lingyi Li
• Lufan Liao
• Fenqiang Zhao
• Tao Zhong
• Yuchen Pei
• Xiangmin Xu

Brain extraction and image quality assessment are two fundamental steps in fetal brain magnetic resonance imaging (MRI) 3D reconstruction and quantification. However, the randomness of fetal position and orientation, the variability of fetal brain morphology, maternal organs around the fetus, and the scarcity of data samples, all add excessive noise and impose a great challenge to automated brain extraction and quality assessment of fetal MRI slices. Conventionally, brain extraction and quality assessment are typically performed independently. However, both of them focus on the brain image representation, so they can be jointly optimized to ensure the network learns more effective features and avoid overfitting. To this end, we propose a novel two-stage dual-task deep learning framework with a brain localization stage and a dual-task stage for joint brain extraction and quality assessment of fetal MRI slices. Specifically, the dual-task module compactly contains a feature extraction module, a quality assessment head and a segmentation head with feature fusion for simultaneous brain extraction and quality assessment. Besides, a transformer architecture is introduced into the feature extraction module and the segmentation head. We utilize a multi-step training strategy to guarantee a stable and successful training of all modules. Finally, we validate our method by a 5-fold cross-validation and ablation study on a dataset with fetal brain MRI slices in different qualities, and perform a cross-dataset validation in addition. Experiments show that the proposed framework achieves very promising performance.

NeurIPS Conference 2024 Conference Paper

CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models

• Peng Xia
• Ze Chen
• Juanxi Tian
• Yangrui Gong
• Ruibo Hou
• Yue Xu
• Zhenbang Wu
• Zhiyuan Fan

Artificial intelligence has significantly impacted medical applications, particularly with the advent of Medical Large Vision Language Models (Med-LVLMs), sparking optimism for the future of automated and personalized healthcare. However, the trustworthiness of Med-LVLMs remains unverified, posing significant risks for future model deployment. In this paper, we introduce CARES and aim to comprehensively evaluate the Trustworthiness of Med-LVLMs across the medical domain. We assess the trustworthiness of Med-LVLMs across five dimensions, including trustfulness, fairness, safety, privacy, and robustness. CARES comprises about 41K question-answer pairs in both closed and open-ended formats, covering 16 medical image modalities and 27 anatomical regions. Our analysis reveals that the models consistently exhibit concerns regarding trustworthiness, often displaying factual inaccuracies and failing to maintain fairness across different demographic groups. Furthermore, they are vulnerable to attacks and demonstrate a lack of privacy awareness. We publicly release our benchmark and code in https: //github. com/richard-peng-xia/CARES.

ICRA Conference 2024 Conference Paper

Crosstalk-Free Impedance-Separating Array Measurement for Iontronic Tactile Sensors

• Funing Hou
• Gang Li
• Chenxing Mu
• Mengqi Shi
• Jixiao Liu
• Shijie Guo

Iontronic tactile sensors are promising to measure spatial-temporal contact information with high performance. However, no suitable measuring method has been presented, due to issues with crosstalk and non-negligible equivalent resistance. Hence, this study presents an impedance-separating method, which does not require complex analog components. A general Quadri-Terminal Impedance Network (QTIN) model is introduced to reduce crosstalk, which has specific compatibility with the impedance-separating method. The precise ranges are measured, showing non-rectangle shapes suitable for the response of iontronic tactile sensors. A simple denoising method is provided to reduce initial array noise obviously. This work could benefit various scenarios, such as human-robot interaction and physiological information monitoring.

JAIR Journal 2024 Journal Article

Detecting Change Intervals with Isolation Distributional Kernel

• Yang Cao
• Ye Zhu
• Kai Ming Ting
• Flora D. Salim
• Hong Xian Li
• Luxing Yang
• Gang Li

Detecting abrupt changes in data distribution is one of the most significant tasks in streaming data analysis. Although many unsupervised Change-Point Detection (CPD) methods have been proposed recently to identify those changes, they still suffer from missing subtle changes, poor scalability, or/and sensitivity to outliers. To meet these challenges, we are the first to generalise the CPD problem as a special case of the Change-Interval Detection (CID) problem. Then we propose a CID method, named iCID, based on a recent Isolation Distributional Kernel (IDK). iCID identifies the change interval if there is a high dissimilarity score between two non-homogeneous temporal adjacent intervals. The data-dependent property and finite feature map of IDK enabled iCID to efficiently identify various types of change-points in data streams with the tolerance of outliers. Moreover, the proposed online and offline versions of iCID have the ability to optimise key parameter settings. The effectiveness and efficiency of iCID have been systematically verified on both synthetic and real-world datasets.

IJCAI Conference 2024 Conference Paper

Detecting Change Intervalswith Isolation Distributional Kernel (Abstract Reprint)

• Yang Cao
• Ye Zhu
• Kai Ming Ting
• Flora D. Salim
• Hong Xian Li
• Luxing Yang
• Gang Li

Detecting abrupt changes in data distribution is one of the most significant tasks in streaming data analysis. Although many unsupervised Change-Point Detection (CPD) methods have been proposed recently to identify those changes, they still suffer from missing subtle changes, poor scalability, or/and sensitivity to outliers. To meet these challenges, we are the first to generalise the CPD problem as a special case of the Change-Interval Detection (CID) problem. Then we propose a CID method, named iCID, based on a recent Isolation Distributional Kernel (IDK). iCID identifies the change interval if there is a high dissimilarity score between two non-homogeneous temporal adjacent intervals. The data-dependent property and finite feature map of IDK enabled iCID to efficiently identify various types of change-points in data streams with the tolerance of outliers. Moreover, the proposed online and offline versions of iCID have the ability to optimise key parameter settings. The effectiveness and efficiency of iCID have been systematically verified on both synthetic and real-world datasets.

EAAI Journal 2024 Journal Article

IDP-Net: Industrial defect perception network based on cross-layer semantic information guidance and context concentration enhancement

• Gang Li
• Shilong Zhao
• Min Li
• Mingle Zhou
• Zuobin Ying

Applications in Engineering: In industry, surface defect detection is crucial for improving product quality. However, there are many challenges in industrial inspection scenarios, such as interference from background noise, complex small-target problems, significant variations in target objects, and the problem of finding a balance between inspection speed and accuracy. To address the above problems, this paper proposes an industrial defect-aware network based on cross-layer semantic information guidance and contextual attention enhancement (IDP-Net). Specifically, IDP-Net has four different new features. The contribution of artificial intelligence: Firstly, to solve the industrial surface context and defect similarity problem, this paper proposes a Lightweight Local Global Feature Extraction Network (LLG-Net), unlike other methods, the effective combination of self-attention blocks and convolution blocks ensures gradual integration of global and local features across multiple layers, to improve the detection ability of targets with significant changes in scale, this paper designs a Multiscale Perceptual Feature Aggregation Network (MPA-Net), adequately fuses the shallow fine-grained information and the deep semantic information. Then, to enhance the connection between multi-scale semantic information, an adaptive cross-layer feature fusion module (ACFF) is proposed, which is novel in integrating the characteristics of multiple adjacent levels to help the model better capture the different scale characterisation of the target. Finally, a Region Attention Module (RAM) is proposed and introduced in the detector to enhance the attention to the critical regions around the target object. In particular, this paper proposes a new localisation loss function (MEIoU) that enhances the network’s attention to objects at different scales. The experimental results show that 94. 3%, 98. 7% and 99. 5% of mAP@. 5 are obtained on steel, PCB and aluminium surface defect datasets, respectively, and 50 FPS is achieved, which is better than the current mainstream detectors and meets the demand of practical industrial production.

EAAI Journal 2024 Journal Article

Landslide displacement prediction with step-like curve based on convolutional neural network coupled with bi-directional gated recurrent unit optimized by attention mechanism

• Shaoqiang Meng
• Zhenming Shi
• Ming Peng
• Gang Li
• Hongchao Zheng
• Liu Liu
• Limin Zhang

This study aims to accurately predict landslide displacement characterized by a step-like displacement curve, resulting from complex interactions among multiple factors, including periodic and variable elements. Specifically, we propose a framework based on a Convolutional Neural Network (CNN) and optimized Bidirectional Gated Recurrent Unit (BiGRU) with an Attention mechanism, designed to forecast landslide displacement with a step-like curve. Initially, landslide displacements are decomposed into periodic and trend terms utilizing complementary ensemble empirical mode decomposition with adaptive noise (CEEMDAN) algorithm. Subsequently, a CNN layer is constructed to extract intricate high-dimensional features, while a BiGRU layer is established to capture temporal dependencies within historical sequences. Notably, an attention mechanism module is introduced to enhance the impact of key information within rainfall, water level, and historical displacement data. Finally, the nonlinear weighted Huber loss function (NLWHL) evaluation method is introduced to assess the accuracy of the model in predicting mutation states. The results demonstrate that the proposed framework exhibited superior accuracy in predicting landslide displacement in comparison to alternative intelligent algorithms. Regarding total displacement, the CNN-BiGRU-Attention model demonstrates superior predictive capabilities, reducing RMSE values by 12. 52%, 14. 15%, 15. 58%, 21. 68%, 21. 84%, 32. 81%, 58. 10%, and 53. 81% in the test set compared to CNN-BiGRU, CNN-BiLSTM, BiGRU-Attention, SMA-GRU, SMA-LSTM, SMA-SVM, GRU, and LSTM, respectively. Furthermore, the introduction of NLWHL underscores the remarkable accuracy of the proposed framework in forecasting both creep and mutation states. Assessing the displacement mutation state provides the necessary opportunity for early detection and intervention.

YNIMG Journal 2024 Journal Article

nBEST: Deep-learning-based non-human primates Brain Extraction and Segmentation Toolbox across ages, sites and species

• Tao Zhong
• Xueyang Wu
• Shujun Liang
• Zhenyuan Ning
• Li Wang
• Yuyu Niu
• Shihua Yang
• Zhuang Kang

Accurate processing and analysis of non-human primate (NHP) brain magnetic resonance imaging (MRI) serves an indispensable role in understanding brain evolution, development, aging, and diseases. Despite the accumulation of diverse NHP brain MRI datasets at various developmental stages and from various imaging sites/scanners, existing computational tools designed for human MRI typically perform poor on NHP data, due to huge differences in brain sizes, morphologies, and imaging appearances across species, sites, and ages, highlighting the imperative for NHP-specialized MRI processing tools. To address this issue, in this paper, we present a robust, generic, and fully automated computational pipeline, called non-human primates Brain Extraction and Segmentation Toolbox (nBEST), whose main functionality includes brain extraction, non-cerebrum removal, and tissue segmentation. Building on cutting-edge deep learning techniques by employing lifelong learning to flexibly integrate data from diverse NHP populations and innovatively constructing 3D U-NeXt architecture, nBEST can well handle structural NHP brain MR images from multi-species, multi-site, and multi-developmental-stage (from neonates to the elderly). We extensively validated nBEST based on, to our knowledge, the largest assemblage dataset in NHP brain studies, encompassing 1,469 scans with 11 species (e.g., rhesus macaques, cynomolgus macaques, chimpanzees, marmosets, squirrel monkeys, etc.) from 23 independent datasets. Compared to alternative tools, nBEST outperforms in precision, applicability, robustness, comprehensiveness, and generalizability, greatly benefiting downstream longitudinal, cross-sectional, and cross-species quantitative analyses. We have made nBEST an open-source toolbox (https://github.com/TaoZhong11/nBEST) and we are committed to its continual refinement through lifelong learning with incoming data to greatly contribute to the research field.

NeurIPS Conference 2024 Conference Paper

PediatricsGPT: Large Language Models as Chinese Medical Assistants for Pediatric Applications

• Dingkang Yang
• Jinjie Wei
• Dongling Xiao
• Shunli Wang
• Tong Wu
• Gang Li
• Mingcheng Li
• Shuaibing Wang

Developing intelligent pediatric consultation systems offers promising prospects for improving diagnostic efficiency, especially in China, where healthcare resources are scarce. Despite recent advances in Large Language Models (LLMs) for Chinese medicine, their performance is sub-optimal in pediatric applications due to inadequate instruction data and vulnerable training procedures. To address the above issues, this paper builds PedCorpus, a high-quality dataset of over 300, 000 multi-task instructions from pediatric textbooks, guidelines, and knowledge graph resources to fulfil diverse diagnostic demands. Upon well-designed PedCorpus, we propose PediatricsGPT, the first Chinese pediatric LLM assistant built on a systematic and robust training pipeline. In the continuous pre-training phase, we introduce a hybrid instruction pre-training mechanism to mitigate the internal-injected knowledge inconsistency of LLMs for medical domain adaptation. Immediately, the full-parameter Supervised Fine-Tuning (SFT) is utilized to incorporate the general medical knowledge schema into the models. After that, we devise a direct following preference optimization to enhance the generation of pediatrician-like humanistic responses. In the parameter-efficient secondary SFT phase, a mixture of universal-specific experts strategy is presented to resolve the competency conflict between medical generalist and pediatric expertise mastery. Extensive results based on the metrics, GPT-4, and doctor evaluations on distinct downstream tasks show that PediatricsGPT consistently outperforms previous Chinese medical LLMs. The project and data will be released at https: //github. com/ydk122024/PediatricsGPT.

NeurIPS Conference 2024 Conference Paper

SpeechForensics: Audio-Visual Speech Representation Learning for Face Forgery Detection

• Yachao Liang
• Min Yu
• Gang Li
• Jianguo Jiang
• Boquan Li
• Feng Yu
• Ning Zhang
• Xiang Meng

Detection of face forgery videos remains a formidable challenge in the field of digital forensics, especially the generalization to unseen datasets and common perturbations. In this paper, we tackle this issue by leveraging the synergy between audio and visual speech elements, embarking on a novel approach through audio-visual speech representation learning. Our work is motivated by the finding that audio signals, enriched with speech content, can provide precise information effectively reflecting facial movements. To this end, we first learn precise audio-visual speech representations on real videos via a self-supervised masked prediction task, which encodes both local and global semantic information simultaneously. Then, the derived model is directly transferred to the forgery detection task. Extensive experiments demonstrate that our method outperforms the state-of-the-art methods in terms of cross-dataset generalization and robustness, without the participation of any fake video in model training.

JBHI Journal 2024 Journal Article

Stimulus-Response Patterns: The Key to Giving Generalizability to Text-Based Depression Detection Models

• Zhenyu Liu
• Yang Wu
• Haibo Zhang
• Gang Li
• Zhijie Ding
• Bin Hu

Text content analysis for depression detection using machine learning techniques has become a prominent area of research. However, previous studies focused mainly on analyzing the textual content, neglecting the fundamental factors driving text generation. Consequently, existing models face the challenge of poor generalization to out-of-domain data as they struggle to capture the crucial features of depression. To address this, we propose a novel computational perspective of “stimulus-response patterns” that brings us closer to the essence of clinical diagnosis of depression. Adopting this computational perspective allows us to conceptually unify diverse datasets and generalize this perspective to common datasets in the field. We introduce the Stimulus-Response Patterns-aware Network (SRP-Net) as an exemplary approach within this computational perspective. To assess the performance of the SRP-Net, we constructed a multi-stimulus dataset and conducted experimental evaluations, demonstrating its exceptional cross-stimulus generalizability. Furthermore, we demonstrated the promising performance of SPR-Net in real medical scenarios and conducted an interpretability analysis of the stimulus-response patterns. Our research investigates the critical role of stimulus-response patterns in enhancing the generalizability of text-based depression detection models, which can potentially facilitate data-driven depression detection to approach the diagnostic accuracy of psychiatrists.

NeurIPS Conference 2024 Conference Paper

UniAR: A Unified model for predicting human Attention and Responses on visual content

• Peizhao Li
• Junfeng He
• Gang Li
• Rachit Bhargava
• Shaolei Shen
• Nachiappan Valliappan
• Youwei Liang
• Hongxiang Gu

Progress in human behavior modeling involves understanding both implicit, early-stage perceptual behavior, such as human attention, and explicit, later-stage behavior, such as subjective preferences or likes. Yet most prior research has focused on modeling implicit and explicit human behavior in isolation; and often limited to a specific type of visual content. We propose UniAR -- a unified model of human attention and preference behavior across diverse visual content. UniAR leverages a multimodal transformer to predict subjective feedback, such as satisfaction or aesthetic quality, along with the underlying human attention or interaction heatmaps and viewing order. We train UniAR on diverse public datasets spanning natural images, webpages, and graphic designs, and achieve SOTA performance on multiple benchmarks across various image domains and behavior modeling tasks. Potential applications include providing instant feedback on the effectiveness of UIs/visual content, and enabling designers and content-creation models to optimize their creation for human-centric improvements.

YNIMG Journal 2023 Journal Article

An attention-based context-informed deep framework for infant brain subcortical segmentation

• Liangjun Chen
• Zhengwang Wu
• Fenqiang Zhao
• Ya Wang
• Weili Lin
• Li Wang
• Gang Li

Precise segmentation of subcortical structures from infant brain magnetic resonance (MR) images plays an essential role in studying early subcortical structural and functional developmental patterns and diagnosis of related brain disorders. However, due to the dynamic appearance changes, low tissue contrast, and tiny subcortical size in infant brain MR images, infant subcortical segmentation is a challenging task. In this paper, we propose a context-guided, attention-based, coarse-to-fine deep framework to precisely segment the infant subcortical structures. At the coarse stage, we aim to directly predict the signed distance maps (SDMs) from multi-modal intensity images, including T1w, T2w, and the ratio of T1w and T2w images, with an SDM-Unet, which can leverage the spatial context information, including the structural position information and the shape information of the target structure, to generate high-quality SDMs. At the fine stage, the predicted SDMs, which encode spatial-context information of each subcortical structure, are integrated with the multi-modal intensity images as the input to a multi-source and multi-path attention Unet (M2A-Unet) for achieving refined segmentation. Both the 3D spatial and channel attention blocks are added to guide the M2A-Unet to focus more on the important subregions and channels. We additionally incorporate the inner and outer subcortical boundaries as extra labels to help precisely estimate the ambiguous boundaries. We validate our method on an infant MR image dataset and on an unrelated neonatal MR image dataset. Compared to eleven state-of-the-art methods, the proposed framework consistently achieves higher segmentation accuracy in both qualitative and quantitative evaluations of infant MR images and also exhibits good generalizability in the neonatal dataset.

EAAI Journal 2023 Journal Article

Deep reinforcement learning for the rapid on-demand design of mechanical metamaterials with targeted nonlinear deformation responses

• Nathan K. Brown
• Anthony P. Garland
• Georges M. Fadel
• Gang Li

Mechanical metamaterials are artificial materials with unique global properties due to the structural geometry and material composition of their unit cell. Typically, mechanical metamaterial unit cells are designed such that, when tessellated, they exhibit unique mechanical properties such as zero or negative Poisson's ratio and negative stiffness. Beyond these applications, mechanical metamaterials can be used to achieve tailorable nonlinear deformation responses. Computational methods such as gradient-based topology optimization (TO) and size/shape optimization (SSO) can be implemented to design these metamaterials. However, both methods can lead to suboptimal solutions or a lack of generalizability. Therefore, this research used deep reinforcement learning (DRL), a subset of deep machine learning that teaches an agent to complete tasks through interactive experiences, to design mechanical metamaterials with specific nonlinear deformation responses in compression or tension. The agent learned to design the unit cells by sequentially adding material to a discrete design domain and being rewarded for achieving the desired deformation response. After training, the agent successfully designed unit cells to exhibit desired deformation responses not experienced during training. This work shows the potential of DRL as a high-level design tool for a wide array of engineering applications.

IJCAI Conference 2023 Conference Paper

Do We Need an Encoder-Decoder to Model Dynamical Systems on Networks?

• Bing Liu
• Wei Luo
• Gang Li
• Jing Huang
• Bo Yang

As deep learning gains popularity in modelling dynamical systems, we expose an underappreciated misunderstanding relevant to modelling dynamics on networks. Strongly influenced by graph neural networks, latent vertex embeddings are naturally adopted in many neural dynamical network models. However, we show that embeddings tend to induce a model that fits observations well but simultaneously has incorrect dynamical behaviours. Recognising that previous studies narrowly focus on short-term predictions during the transient phase of a flow, we propose three tests for correct long-term behaviour, and illustrate how an embedding-based dynamical model fails these tests, and analyse the causes, particularly through the lens of topological conjugacy. In doing so, we show that the difficulties can be avoided by not using embedding. We propose a simple embedding-free alternative based on parametrising two additive vector-field components. Through extensive experiments, we verify that the proposed model can reliably recover a broad class of dynamics on different network topologies from time series data.

EAAI Journal 2023 Journal Article

ICA-Net: Industrial defect detection network based on convolutional attention guidance and aggregation of multiscale features

• Shilong Zhao
• Gang Li
• Mingle Zhou
• Min Li

Detecting surface defects in the industry is essential for improving the quality of industrial products and maintaining product safety. However, problems such as the similarity of defects, significant variation in the scale of the target object, and the balance between detection speed and accuracy in industrial inspection scenarios have been considerable research topics in this field. This paper proposes an industrial defect detection network based on convolutional attention-guided and aggregated multiscale features to address these issues (ICA-Net). Firstly, for similarity defects in complex backgrounds, this paper proposes a backbone network with a combination of lightweight convolutional blocks and self-attentive modules to fully extract images’ local and global information and enhance the network’s expressiveness. Secondly, to make full use of the shallow fine-grained features and deep semantic features of the backbone network to improve the detection capability of defects with significant scale changes, this paper designs a cross-layer multiscale feature fusion network (CEF-Net), which fully fuses the features of adjacent layers and cross-layers through a reweighting feature strategy to enrich the network feature transfer path and ensure the efficient fusion of different scale features in the network. At the same time, the fine-grained feature fusion module (FFM) is used to fuse elements from multiple layers to extract more contextual information, enhance the extraction of fine-grained features and improve the detection capability of complex small targets. Finally, to address the problems of inaccurate regression localization and low detection accuracy of defects in existing industrial algorithms, a new IoU loss function (G-IOU) is proposed for regressing the intersection part of the predicted frame and the actual structure according to the aspect ratio of the real frame during the model regression to improve the accuracy and stability of detection. The experimental results show that 94. 1%, 98. 6%, 99. 4%, 98. 8% and 96. 5% of mAP@. 5 are obtained on steel, PCB, aluminium, automobile and Xsteel steel metal surface defect datasets, respectively, and 48 FPS is achieved, which is superior to the current mainstream detectors and meets the needs of practical industrial production.

EAAI Journal 2023 Journal Article

IDD-Net: Industrial defect detection method based on Deep-Learning

• Zekai Zhang
• Mingle Zhou
• Honglin Wan
• Min Li
• Gang Li
• Delong Han

Detecting defects in industrial products is one of the most widespread applications of industrial automation. Various product defects, large similarities, and drastic changes in scale in industrial scenarios pose challenges to existing industrial inspection networks. This paper proposes a deep learning-based industrial defect detection method (IDD-Net) to address the above challenges. Specifically, IDD-Net has three distinct features. First, for the defects of diversity and similarity (rolled-in_scale, crazing in steel defects), IDD-Net designed a novel local–global backbone feature network (LGB-Net). Second, IDD-Net proposes a novel Three-Layer Feature Aggregation network (TFLA-Net) to solve the problem of drastic scale changes. TFLA-Net adopts a novel three-layer descending method to aggregate semantic and fine-grained features effectively. At the same time, the dense connection of adjacent nodes of TFLA-Net ensures the efficient fusion of features of different scales in the network. In particular, this paper proposes a novel IoU loss (Defect-IoU loss) for the problem of object loss deviation at different scales. The novelty of Defect-IoU Loss is that the loss value is scaled by the difference in the area of different scale objects, which is more conducive to the balance of multi-scale object loss. The experimental results show that the calculation amount of IDD-Net is only 24. 9 Gflops, and the mAP@. 5 of 79. 66%, 99. 5%, and 95. 9% in the steel defect, aluminium defect, and PCB defect datasets were respectively obtained, surpassing all comparison models. In addition, the test in the actual industrial scene also demonstrates the feasibility of the application of IDD-Net.

NeurIPS Conference 2023 Conference Paper

Maximization of Average Precision for Deep Learning with Adversarial Ranking Robustness

• Gang Li
• Wei Tong
• Tianbao Yang

This paper seeks to address a gap in optimizing Average Precision (AP) while ensuring adversarial robustness, an area that has not been extensively explored to the best of our knowledge. AP maximization for deep learning has widespread applications, particularly when there is a significant imbalance between positive and negative examples. Although numerous studies have been conducted on adversarial training, they primarily focus on robustness concerning accuracy, ensuring that the average accuracy on adversarially perturbed examples is well maintained. However, this type of adversarial robustness is insufficient for many applications, as minor perturbations on a single example can significantly impact AP while not greatly influencing the accuracy of the prediction system. To tackle this issue, we introduce a novel formulation that combines an AP surrogate loss with a regularization term representing adversarial ranking robustness, which maintains the consistency between ranking of clean data and that of perturbed data. We then devise an efficient stochastic optimization algorithm to optimize the resulting objective. Our empirical studies, which compare our method to current leading adversarial training baselines and other robust AP maximization strategies, demonstrate the effectiveness of the proposed approach. Notably, our methods outperform a state-of-the-art method (TRADES) by more than 4\% in terms of robust AP against PGD attacks while achieving 7\% higher AP on clean data simultaneously on CIFAR10 and CIFAR100. The code is available at: https: //github. com/GangLii/Adversarial-AP

JBHI Journal 2023 Journal Article

The Human Activity Radar Challenge: Benchmarking Based on the ‘Radar Signatures of Human Activities’ Dataset From Glasgow University

• Shufan Yang
• Julien Le Kernec
• Olivier Romain
• Francesco Fioranelli
• Pierre Cadart
• Jérémy Fix
• Chenfang Ren
• Giovanni Manfredi

Radar is an extremely valuable sensing technology for detecting moving targets and measuring their range, velocity, and angular positions. When people are monitored at home, radar is more likely to be accepted by end-users, as they already use WiFi, is perceived as privacy-preserving compared to cameras, and does not require user compliance as wearable sensors do. Furthermore, it is not affected by lighting conditions nor requires artificial lights that could cause discomfort in the home environment. So, radar-based human activities classification in the context of assisted living can empower an aging society to live at home independently longer. However, challenges remain as to the formulation of the most effective algorithms for radar-based human activities classification and their validation. To promote the exploration and cross-evaluation of different algorithms, our dataset released in 2019 was used to benchmark various classification approaches. The challenge was open from February 2020 to December 2020. A total of 23 organizations worldwide, forming 12 teams from academia and industry, participated in the inaugural Radar Challenge, and submitted 188 valid entries to the challenge. This paper presents an overview and evaluation of the approaches used for all primary contributions in this inaugural challenge. The proposed algorithms are summarized, and the main parameters affecting their performances are analyzed.

YNIMG Journal 2022 Journal Article

A 4D infant brain volumetric atlas based on the UNC/UMN baby connectome project (BCP) cohort

• Liangjun Chen
• Zhengwang Wu
• Dan Hu
• Ya Wang
• Fenqiang Zhao
• Tao Zhong
• Weili Lin
• Li Wang

Spatiotemporal (four-dimensional) infant-dedicated brain atlases are essential for neuroimaging analysis of early dynamic brain development. However, due to the substantial technical challenges in the acquisition and processing of infant brain MR images, 4D atlases densely covering the dynamic brain development during infancy are still scarce. Few existing ones generally have fuzzy tissue contrast and low spatiotemporal resolution, leading to degraded accuracy of atlas-based normalization and subsequent analyses. To address this issue, in this paper, we construct a 4D structural MRI atlas for infant brains based on the UNC/UMN Baby Connectome Project (BCP) dataset, which features a high spatial resolution, extensive age-range coverage, and densely sampled time points. Specifically, 542 longitudinal T1w and T2w scans from 240 typically developing infants up to 26-month of age were utilized for our atlas construction. To improve the co-registration accuracy of the infant brain images, which typically exhibit dynamic appearance with low tissue contrast, we employed the state-of-the-art registration method and leveraged our generated reliable brain tissue probability maps in addition to the intensity images to improve the alignment of individual images. To achieve consistent region labeling on both infant and adult brain images for facilitating region-based analysis across ages, we mapped the widely used Desikan cortical parcellation onto our atlas by following an age-decreasing mapping manner. Meanwhile, the typical subcortical structures were manually delineated to facilitate the studies related to the subcortex. Compared with the existing infant brain atlases, our 4D atlas has much higher spatiotemporal resolution and preserves more structural details, and thus can boost accuracy in neurodevelopmental analysis during infancy.

NeurIPS Conference 2022 Conference Paper

DTG-SSOD: Dense Teacher Guidance for Semi-Supervised Object Detection

• Gang Li
• Xiang Li
• Yujie Wang
• Wu Yichao
• Ding Liang
• Shanshan Zhang

The Mean-Teacher (MT) scheme is widely adopted in semi-supervised object detection (SSOD). In MT, sparse pseudo labels, offered by the final predictions of the teacher (e. g. , after Non Maximum Suppression (NMS) post-processing), are adopted for the dense supervision for the student via hand-crafted label assignment. However, the "sparse-to-dense'' paradigm complicates the pipeline of SSOD, and simultaneously neglects the powerful direct, dense teacher supervision. In this paper, we attempt to directly leverage the dense guidance of teacher to supervise student training, i. e. , the "dense-to-dense'' paradigm. Specifically, we propose the Inverse NMS Clustering (INC) and Rank Matching (RM) to instantiate the dense supervision, without the widely used, conventional sparse pseudo labels. INC leads the student to group candidate boxes into clusters in NMS as the teacher does, which is implemented by learning grouping information revealed in NMS procedure of the teacher. After obtaining the same grouping scheme as the teacher via INC, the student further imitates the rank distribution of the teacher over clustered candidates through Rank Matching. With the proposed INC and RM, we integrate Dense Teacher Guidance into Semi-Supervised Object Detection (termed "DTG-SSOD''), successfully abandoning sparse pseudo labels and enabling more informative learning on unlabeled data. On COCO benchmark, our DTG-SSOD achieves state-of-the-art performance under various labelling ratios. For example, under 10% labelling ratio, DTG-SSOD improves the supervised baseline from 26. 9 to 35. 9 mAP, outperforming the previous best method Soft Teacher by 1. 9 points.

EAAI Journal 2022 Journal Article

Fault diagnosis of modular multilevel converter based on adaptive chirp mode decomposition and temporal convolutional network

• Qun Guo
• Xinhao Zhang
• Jing Li
• Gang Li

The reliability of the insulated gate bipolar transistors (IGBTs) is essential to the stable operation of the modular multilevel converter (MMC) system. However, there are a large number of IGBTs in the MMC system and the open-circuit faults of IGBTs are usually so hidden that it is difficult to find. Therefore, this article proposes a fault diagnosis framework based on temporal convolutional network (TCN) integrating adaptive chirp mode decomposition (ACMD) and silhouette coefficient (SC). First, ACMD is used to extract and reconstruct signal components from the original signal. Then, in order to avoid artificial selection of signal components, silhouette coefficient is introduced to characterize the importance of each component. Finally, the TCN model automatically extracts the features of the signal components and outputs the classification results. The main contributions are as follows: (1) A complete fault diagnosis framework that can adaptively extract features and perform fault classification is proposed in the paper. (2) For the MMC using the carrier-phase-shifted pulsewidth modulation strategy, the fault can be located to the IGBT by the output current. (3) Under certain noise conditions, the fault diagnosis proposed in the paper method still has good robustness. (4) The signal visualization of different residual blocks and channels explains the working mechanism of the AMCD-SC-TCN framework.

AAAI Conference 2022 Conference Paper

Knowledge Distillation for Object Detection via Rank Mimicking and Prediction-Guided Feature Imitation

• Gang Li
• Xiang Li
• Yujie Wang
• Shanshan Zhang
• Yichao Wu
• Ding Liang

Knowledge Distillation (KD) is a widely-used technology to inherit information from cumbersome teacher models to compact student models, consequently realizing model compression and acceleration. Compared with image classification, object detection is a more complex task, and designing specific KD methods for object detection is non-trivial. In this work, we elaborately study the behaviour difference between the teacher and student detection models, and obtain two intriguing observations: First, the teacher and student rank their detected candidate boxes quite differently, which results in their precision discrepancy. Second, there is a considerable gap between the feature response differences and prediction differences between teacher and student, indicating that equally imitating all the feature maps of the teacher is the sub-optimal choice for improving the student’s accuracy. Based on the two observations, we propose Rank Mimicking (RM) and Prediction-guided Feature Imitation (PFI) for distilling one-stage detectors, respectively. RM takes the rank of candidate boxes from teachers as a new form of knowledge to distill, which consistently outperforms the traditional soft label distillation. PFI attempts to correlate feature differences with prediction differences, making feature imitation directly help to improve the student’s accuracy. On MS COCO and PASCAL VOC benchmarks, extensive experiments are conducted on various detectors with different backbones to validate the effectiveness of our method. Specifically, RetinaNet with ResNet50 achieves 40. 4% mAP on MS COCO, which is 3. 5% higher than its baseline, and also outperforms previous KD methods.

YNIMG Journal 2022 Journal Article

Longitudinal brain atlases of early developing cynomolgus macaques from birth to 48 months of age

• Tao Zhong
• Jingkuan Wei
• Kunhua Wu
• Liangjun Chen
• Fenqiang Zhao
• Yuchen Pei
• Ya Wang
• Hongjiang Zhang

Longitudinal brain imaging atlases with densely sampled time-points and ancillary anatomical information are of fundamental importance in studying early developmental characteristics of human and non-human primate brains during infancy, which feature extremely dynamic imaging appearance, brain shape and size. However, for non-human primates, which are highly valuable animal models for understanding human brains, the existing brain atlases are mainly developed based on adults or adolescents, denoting a notable lack of temporally densely-sampled atlases covering the dynamic early brain development. To fill this critical gap, in this paper, we construct a comprehensive set of longitudinal brain atlases and associated tissue probability maps (gray matter, white matter, and cerebrospinal fluid) with totally 12 time-points from birth to 4 years of age (i.e., 1, 2, 3, 4, 5, 6, 9, 12, 18, 24, 36, and 48 months of age) based on 175 longitudinal structural MRI scans from 39 typically-developing cynomolgus macaques, by leveraging state-of-the-art computational techniques tailored for early developing brains. Furthermore, to facilitate region-based analysis using our atlases, we also provide two popular hierarchy parcellations, i.e., cortical hierarchy maps (6 levels) and subcortical hierarchy maps (6 levels), on our longitudinal macaque brain atlases. These early developing atlases, which have the densest time-points during infancy (to the best of our knowledge), will greatly facilitate the studies of macaque brain development.

NeurIPS Conference 2022 Conference Paper

Multi-block-Single-probe Variance Reduced Estimator for Coupled Compositional Optimization

• Wei Jiang
• Gang Li
• Yibo Wang
• Lijun Zhang
• Tianbao Yang

Variance reduction techniques such as SPIDER/SARAH/STORM have been extensively studied to improve the convergence rates of stochastic non-convex optimization, which usually maintain and update a sequence of estimators for a single function across iterations. What if we need to track multiple functional mappings across iterations but only with access to stochastic samples of $\mathcal{O}(1)$ functional mappings at each iteration? There is an important application in solving an emerging family of coupled compositional optimization problems in the form of $\sum_{i=1}^m f_i(g_i(\mathbf{w}))$, where $g_i$ is accessible through a stochastic oracle. The key issue is to track and estimate a sequence of $\mathbf g(\mathbf{w})=(g_1(\mathbf{w}), \ldots, g_m(\mathbf{w}))$ across iterations, where $\mathbf g(\mathbf{w})$ has $m$ blocks and it is only allowed to probe $\mathcal{O}(1)$ blocks to attain their stochastic values and Jacobians. To improve the complexity for solving these problems, we propose a novel stochastic method named Multi-block-Single-probe Variance Reduced (MSVR) estimator to track the sequence of $\mathbf g(\mathbf{w})$. It is inspired by STORM but introduces a customized error correction term to alleviate the noise not only in stochastic samples for the selected blocks but also in those blocks that are not sampled. With the help of the MSVR estimator, we develop several algorithms for solving the aforementioned compositional problems with improved complexities across a spectrum of settings with non-convex/convex/strongly convex/Polyak-Lojasiewicz (PL) objectives. Our results improve upon prior ones in several aspects, including the order of sample complexities and dependence on the strong convexity parameter. Empirical studies on multi-task deep AUC maximization demonstrate the better performance of using the new estimator.

JBHI Journal 2022 Journal Article

Multimodal Biosensing for Vestibular Network-Based Cybersickness Detection

• Gang Li
• Mark McGill
• Stephen Brewster
• Chao Ping Chen
• Joaquin A. Anguera
• Adam Gazzaley
• Frank Pollick

Virtual reality (VR) has the potential to induce cybersickness (CS), which impedes CS-susceptible VR users from the benefit of emerging VR applications. To better detect CS, the current study investigated whether/how the newly proposed human vestibular network (HVN) is involved in flagship consumer VR-induced CS by simultaneously recording autonomic physiological signals as well as neural signals generated in sensorimotor and cognitive domains. The VR stimuli were made up of one or two moderate CS-inducing entertaining task(s) as well as a mild CS-inducing cognitive task implemented before and after the moderate CS task(s). Results not only showed that CS impaired cognitive control ability, represented by the degree of attentional engagement, but also revealed that combined indicators from all three HVN domains could together establish the best regression relationship with CS ratings. More importantly, we found that every HVN domain had its unique advantage with the dynamic changes in CS severity and time. These results provide evidence for involvement of the HVN in CS and indicate the necessity of HVN-based CS detection.

NeurIPS Conference 2022 Conference Paper

SemMAE: Semantic-Guided Masking for Learning Masked Autoencoders

• Gang Li
• Heliang Zheng
• Daqing Liu
• Chaoyue Wang
• Bing Su
• Changwen Zheng

Recently, significant progress has been made in masked image modeling to catch up to masked language modeling. However, unlike words in NLP, the lack of semantic decomposition of images still makes masked autoencoding (MAE) different between vision and language. In this paper, we explore a potential visual analogue of words, i. e. , semantic parts, and we integrate semantic information into the training process of MAE by proposing a Semantic-Guided Masking strategy. Compared to widely adopted random masking, our masking strategy can gradually guide the network to learn various information, i. e. , from intra-part patterns to inter-part relations. In particular, we achieve this in two steps. 1) Semantic part learning: we design a self-supervised part learning method to obtain semantic parts by leveraging and refining the multi-head attention of a ViT-based encoder. 2) Semantic-guided MAE (SemMAE) training: we design a masking strategy that varies from masking a portion of patches in each part to masking a portion of (whole) parts in an image. Extensive experiments on various vision tasks show that SemMAE can learn better image representation by integrating semantic information. In particular, SemMAE achieves 84. 5% fine-tuning accuracy on ImageNet-1k, which outperforms the vanilla MAE by 1. 4%. In the semantic segmentation and fine-grained recognition tasks, SemMAE also brings significant improvements and yields the state-of-the-art performance.

ICML Conference 2022 Conference Paper

When AUC meets DRO: Optimizing Partial AUC for Deep Learning with Non-Convex Convergence Guarantee

• Dixian Zhu
• Gang Li
• Bokun Wang
• Xiaodong Wu
• Tianbao Yang

In this paper, we propose systematic and efficient gradient-based methods for both one-way and two-way partial AUC (pAUC) maximization that are applicable to deep learning. We propose new formulations of pAUC surrogate objectives by using the distributionally robust optimization (DRO) to define the loss for each individual positive data. We consider two formulations of DRO, one of which is based on conditional-value-at-risk (CVaR) that yields a non-smooth but exact estimator for pAUC, and another one is based on a KL divergence regularized DRO that yields an inexact but smooth (soft) estimator for pAUC. For both one-way and two-way pAUC maximization, we propose two algorithms and prove their convergence for optimizing their two formulations, respectively. Experiments demonstrate the effectiveness of the proposed algorithms for pAUC maximization for deep learning on various datasets.

YNIMG Journal 2021 Journal Article

DIKA-Nets: Domain-invariant knowledge-guided attention networks for brain skull stripping of early developing macaques

• Tao Zhong
• Fenqiang Zhao
• Yuchen Pei
• Zhenyuan Ning
• Lufan Liao
• Zhengwang Wu
• Yuyu Niu
• Li Wang

As non-human primates, macaques have a close phylogenetic relationship to human beings and have been proven to be a valuable and widely used animal model in human neuroscience research. Accurate skull stripping (aka. brain extraction) of brain magnetic resonance imaging (MRI) is a crucial prerequisite in neuroimaging analysis of macaques. Most of the current skull stripping methods can achieve satisfactory results for human brains, but when applied to macaque brains, especially during early brain development, the results are often unsatisfactory. In fact, the early dynamic, regionally-heterogeneous development of macaque brains, accompanied by poor and age-related contrast between different anatomical structures, poses significant challenges for accurate skull stripping. To overcome these challenges, we propose a fully-automated framework to effectively fuse the age-specific intensity information and domain-invariant prior knowledge as important guiding information for robust skull stripping of developing macaques from 0 to 36 months of age. Specifically, we generate Signed Distance Map (SDM) and Center of Gravity Distance Map (CGDM) based on the intermediate segmentation results as guidance. Instead of using local convolution, we fuse all information using the Dual Self-Attention Module (DSAM), which can capture global spatial and channel-dependent information of feature maps. To extensively evaluate the performance, we adopt two relatively-large challenging MRI datasets from rhesus macaques and cynomolgus macaques, respectively, with a total of 361 scans from two different scanners with different imaging protocols. We perform cross-validation by using one dataset for training and the other one for testing. Our method outperforms five popular brain extraction tools and three deep-learning-based methods on cross-source MRI datasets without any transfer learning.

I&C Journal 2021 Journal Article

Extremal synchronizing circular automata

• Yong He
• Xueping Chen
• Gang Li
• Shiyuan Sun

An n-state synchronizing automaton is said to be extremal if it has the reset threshold ( n − 1 ) 2. An extremal synchronizing automaton is specially called an extreme synchronizing automaton if it is no longer an extremal synchronizing automaton after the removal of at least one letter. The Černý automata provide an infinite sequence of extreme synchronizing automata. Besides this, up to isomorphism, only eight isolated examples of extreme synchronizing automata on at least three states have been found. Since the Černý automata and one of the eight isolated examples are circular, one may say that almost all known extreme synchronizing automata are circular. In 2006, Trahtman conjectured that no other extreme synchronizing automaton on at least three states exists. In this paper, all extremal synchronizing circular automata and all extreme synchronizing circular automata are determined. As a consequence, Trahtman's conjecture is confirmed for circular automata.

NeurIPS Conference 2021 Conference Paper

Learnable Fourier Features for Multi-dimensional Spatial Positional Encoding

• Yang Li
• Si Si
• Gang Li
• Cho-Jui Hsieh
• Samy Bengio

Attentional mechanisms are order-invariant. Positional encoding is a crucial component to allow attention-based deep model architectures such as Transformer to address sequences or images where the position of information matters. In this paper, we propose a novel positional encoding method based on learnable Fourier features. Instead of hard-coding each position as a token or a vector, we represent each position, which can be multi-dimensional, as a trainable encoding based on learnable Fourier feature mapping, modulated with a multi-layer perceptron. The representation is particularly advantageous for a spatial multi-dimensional position, e. g. , pixel positions on an image, where $L_2$ distances or more complex positional relationships need to be captured. Our experiments based on several public benchmark tasks show that our learnable Fourier feature representation for multi-dimensional positional encoding outperforms existing methods by both improving the accuracy and allowing faster convergence.

YNIMG Journal 2021 Journal Article

The maturation and cognitive relevance of structural brain network organization from early infancy to childhood

• Mackenzie Woodburn
• Cheyenne L. Bricken
• Zhengwang Wu
• Gang Li
• Li Wang
• Weili Lin
• Margaret A. Sheridan
• Jessica R. Cohen

The interactions of brain regions with other regions at the network level likely provide the infrastructure necessary for cognitive processes to develop. Specifically, it has been theorized that in infancy brain networks become more modular, or segregated, to support early cognitive specialization, before integration across networks increases to support the emergence of higher-order cognition. The present study examined the maturation of structural covariance networks (SCNs) derived from longitudinal cortical thickness data collected between infancy and childhood (0-6 years). We assessed modularity as a measure of network segregation and global efficiency as a measure of network integration. At the group level, we observed trajectories of increasing modularity and decreasing global efficiency between early infancy and six years. We further examined subject-based maturational coupling networks (sbMCNs) in a subset of this cohort with cognitive outcome data at 8-10 years, which allowed us to relate the network organization of longitudinal cortical thickness maturation to cognitive outcomes in middle childhood. We found that lower global efficiency of sbMCNs throughout early development (across the first year) related to greater motor learning at 8-10 years. Together, these results provide novel evidence characterizing the maturation of brain network segregation and integration across the first six years of life, and suggest that specific trajectories of brain network maturation contribute to later cognitive outcomes.

JBHI Journal 2020 Journal Article

Canonical Correlation Analysis of Imaging Genetics Data Based on Statistical Independence and Structural Sparsity

• Yipu Zhang
• Peng Peng
• Yongfeng Ju
• Gang Li
• Vince D. Calhoun
• Yu-Ping Wang

Current developments of neuroimaging and genetics promote an integrative and compressive study of schizophrenia. However, it is still difficult to explore how gene mutations are related to brain abnormalities due to the high dimension but low sample size of these data. Conventional approaches reduce the dimension of dataset separately and then calculate the correlation, but ignore the effects of the response variables and the structure of data. To improve the identification of risk genes and abnormal brain regions on schizophrenia, in this paper, we propose a novel method called Independence and Structural sparsity Canonical Correlation Analysis (ISCCA). ISCCA combines independent component analysis (ICA) and Canonical Correlation Analysis (CCA) to reduce the collinear effects, which also incorporate graph structure of the data into the model to improve the accuracy of feature selection. The results from simulation studies demonstrate its higher accuracy in discovering correlations compared with other competing methods. Moreover, applying ISCCA to a real imaging genetics dataset collected by Mind Clinical Imaging Consortium (MCIC), a set of distinct gene-ROI interactions are identified, which are verified to be both statistically and biologically significant.

JBHI Journal 2020 Journal Article

Hierarchical Rough-to-Fine Model for Infant Age Prediction Based on Cortical Features

• Dan Hu
• Zhengwang Wu
• Weili Lin
• Gang Li
• Dinggang Shen

Prediction of the chronological age based on neuroimaging data is important for brain development analysis and brain disease diagnosis. Although many researches have been conducted for age prediction of older children and adults, little work has been dedicated to infants. To this end, this paper focuses on predicting infant age from birth to 2-year old using brain MR images, as well as identifying some related biomarkers. However, brain development during infancy is too rapid and heterogeneous to be accurately modeled by the conventional regression models. To address this issue, a two-stage prediction method is proposed. Specifically, our method first roughly predicts the age range of an infant and then finely predicts the accurate chronological age based on a learned, age-group-specific regression model. Combining this two-stage prediction method with another complementary one-stage prediction method, a hierarchical rough-to-fine (HRtoF) model is built. HRtoF effectively splits the rapid and heterogeneous changes during a long time period into several short time ranges and further mines the discrimination capability of cortical features, thus reaching high accuracy in infant age prediction. Taking 8 types of cortical morphometric features from structural MRI as predictors, the effectiveness of our proposed HRtoF model is validated using an infant dataset including 50 healthy subjects with 251 longitudinal MRI scans from 14 to 797 days. Comparing with five state-of-the-art regression methods, HRtoF model reduces the mean absolute error of the prediction from >48 days to 32. 1 days. The correlation coefficient of the predicted age and the chronological age reaches 0. 963. Moreover, based on HRtoF, the relative contributions of the eight types of cortical features for age prediction are also studied.

AAAI Conference 2020 Conference Paper

Sparsity-Inducing Binarized Neural Networks

• Peisong Wang
• Xiangyu He
• Gang Li
• Tianli Zhao
• Jian Cheng

Binarization of feature representation is critical for Binarized Neural Networks (BNNs). Currently, sign function is the commonly used method for feature binarization. Although it works well on small datasets, the performance on ImageNet remains unsatisfied. Previous methods mainly focus on minimizing quantization error, improving the training strategies and decomposing each convolution layer into several binary convolution modules. However, whether sign is the only option for binarization has been largely overlooked. In this work, we propose the Sparsity-inducing Binarized Neural Network (Si-BNN), to quantize the activations to be either 0 or +1, which introduces sparsity into binary representation. We further introduce trainable thresholds into the backward function of binarization to guide the gradient propagation. Our method dramatically outperforms current state-ofthe-arts, lowering the performance gap between full-precision networks and BNNs on mainstream architectures, achieving the new state-of-the-art on binarized AlexNet (Top-1 50. 5%), ResNet-18 (Top-1 59. 7%), and VGG-Net (Top-1 63. 2%). At inference time, Si-BNN still enjoys the high efficiency of exclusive-not-or (xnor) operations.

YNIMG Journal 2019 Journal Article

Computational neuroanatomy of baby brains: A review

• Gang Li
• Li Wang
• Pew-Thian Yap
• Fan Wang
• Zhengwang Wu
• Yu Meng
• Pei Dong
• Jaeil Kim

The first postnatal years are an exceptionally dynamic and critical period of structural, functional and connectivity development of the human brain. The increasing availability of non-invasive infant brain MR images provides unprecedented opportunities for accurate and reliable charting of dynamic early brain developmental trajectories in understanding normative and aberrant growth. However, infant brain MR images typically exhibit reduced tissue contrast (especially around 6 months of age), large within-tissue intensity variations, and regionally-heterogeneous, dynamic changes, in comparison with adult brain MR images. Consequently, the existing computational tools developed typically for adult brains are not suitable for infant brain MR image processing. To address these challenges, many infant-tailored computational methods have been proposed for computational neuroanatomy of infant brains. In this review paper, we provide a comprehensive review of the state-of-the-art computational methods for infant brain MRI processing and analysis, which have advanced our understanding of early postnatal brain development. We also summarize publically available infant-dedicated resources, including MRI datasets, computational tools, grand challenges, and brain atlases. Finally, we discuss the limitations in current research and suggest potential future research directions.

YNIMG Journal 2019 Journal Article

Exploring folding patterns of infant cerebral cortex based on multi-view curvature features: Methods and applications

• Dingna Duan
• Shunren Xia
• Islem Rekik
• Yu Meng
• Zhengwang Wu
• Li Wang
• Weili Lin
• John H. Gilmore

The highly convoluted cortical folding of the human brain is intriguingly complex and variable across individuals. Exploring the underlying representative patterns of cortical folding is of great importance for many neuroimaging studies. At term birth, all major cortical folds are established and are minimally affected by the complicated postnatal environments; hence, neonates are the ideal candidates for exploring early postnatal cortical folding patterns, which yet remain largely unexplored. In this paper, we propose a novel method for exploring the representative regional folding patterns of infant brains. Specifically, first, multi-view curvature features are constructed to comprehensively characterize the complex characteristics of cortical folding. Second, for each view of curvature features, a similarity matrix is computed to measure the similarity of cortical folding in a specific region between any pair of subjects. Next, a similarity network fusion method is adopted to nonlinearly and adaptively fuse all the similarity matrices into a single one for retaining both shared and complementary similarity information of the multiple characteristics of cortical folding. Finally, based on the fused similarity matrix and a hierarchical affinity propagation clustering approach, all subjects are automatically grouped into several clusters to obtain the representative folding patterns. To show the applications, we have applied the proposed method to a large-scale dataset with 595 normal neonates and discovered representative folding patterns in several cortical regions, i. e. , the superior temporal gyrus (STG), inferior frontal gyrus (IFG), precuneus, and cingulate cortex. Meanwhile, we have revealed sex difference in STG, IFG, and cingulate cortex, as well as hemispheric asymmetries in STG and cingulate cortex in terms of cortical folding patterns. Moreover, we have also validated the proposed method on a public adult dataset, i. e. , the Human Connectome Project (HCP), and revealed that certain major cortical folding patterns of adults are largely established at term birth.

YNIMG Journal 2019 Journal Article

First-year development of modules and hubs in infant brain functional networks

• Xuyun Wen
• Han Zhang
• Gang Li
• Mingxia Liu
• Weiyan Yin
• Weili Lin
• Jun Zhang
• Dinggang Shen

The human brain develops rapidly in the first postnatal year, in which rewired functional brain networks could shape later behavioral and cognitive performance. Resting-state functional magnetic resonances imaging (rs-fMRI) and complex network analysis have been widely used for characterizing the developmental brain functional connectome. Yet, such studies focusing on the first year of postnatal life are still very limited. Leveraging normally developing longitudinal infant rs-fMRI scans from neonate to one year of age, we investigated how brain functional networks develop at a fine temporal scale (every 3 months). Considering challenges in the infant fMRI-based network analysis, we developed a novel algorithm to construct the robust, temporally consistent and modular structure augmented group-level network based on which functional modules were detected at each age. Our study reveals that the brain functional network is gradually subdivided into an increasing number of functional modules accompanied by the strengthened intra- and inter-modular connectivities. Based on the developing modules, we found connector hubs (the high-centrality regions connecting different modules) emerging and increasing, while provincial hubs (the high-centrality regions connecting regions in the same module) diminishing. Further region-wise longitudinal analysis validates that different hubs have distinct developmental trajectories of the intra- and inter-modular connections suggesting different types of role transitions in network, such as non-hubs to hubs or provincial hubs to connector hubs et al. All findings indicate that functional segregation and integration are both increased in the first year of postnatal life. The module reorganization and hub transition lead to more efficient brain networks, featuring increasingly segregated modular structure and more connector hubs. This study provides the first comprehensive report of the development of functional brain networks at a 3-month interval throughout the first postnatal year of life, which provides essential information to the future neurodevelopmental and developmental disorder studies.

YNIMG Journal 2019 Journal Article

Multi-task prediction of infant cognitive scores from longitudinal incomplete neuroimaging data

• Ehsan Adeli
• Yu Meng
• Gang Li
• Weili Lin
• Dinggang Shen

Early postnatal brain undergoes a stunning period of development. Over the past few years, research on dynamic infant brain development has received increased attention, exhibiting how important the early stages of a child's life are in terms of brain development. To precisely chart the early brain developmental trajectories, longitudinal studies with data acquired over a long-enough period of infants' early life is essential. However, in practice, missing data from different time point(s) during the data gathering procedure is often inevitable. This leads to incomplete set of longitudinal data, which poses a major challenge for such studies. In this paper, prediction of multiple future cognitive scores with incomplete longitudinal imaging data is modeled into a multi-task machine learning framework. To efficiently learn this model, we account for selection of informative features (i. e. , neuroimaging morphometric measurements for different time points), while preserving the structural information and the interrelation between these multiple cognitive scores. Several experiments are conducted on a carefully acquired in-house dataset, and the results affirm that we can predict the cognitive scores measured at the age of four years old, using the imaging data of earlier time points, as early as 24 months of age, with a reasonable performance (i. e. , root mean square error of 0. 18).

YNIMG Journal 2019 Journal Article

The UNC/UMN Baby Connectome Project (BCP): An overview of the study design and protocol development

• Brittany R. Howell
• Martin A. Styner
• Wei Gao
• Pew-Thian Yap
• Li Wang
• Kristine Baluyot
• Essa Yacoub
• Geng Chen

The human brain undergoes extensive and dynamic growth during the first years of life. The UNC/UMN Baby Connectome Project (BCP), one of the Lifespan Connectome Projects funded by NIH, is an ongoing study jointly conducted by investigators at the University of North Carolina at Chapel Hill and the University of Minnesota. The primary objective of the BCP is to characterize brain and behavioral development in typically developing infants across the first 5 years of life. The ultimate goals are to chart emerging patterns of structural and functional connectivity during this period, map brain-behavior associations, and establish a foundation from which to further explore trajectories of health and disease. To accomplish these goals, we are combining state of the art MRI acquisition and analysis techniques, including high-resolution structural MRI (T1-and T2-weighted images), diffusion imaging (dMRI), and resting state functional connectivity MRI (rfMRI). While the overall design of the BCP largely is built on the protocol developed by the Lifespan Human Connectome Project (HCP), given the unique age range of the BCP cohort, additional optimization of imaging parameters and consideration of an age appropriate battery of behavioral assessments were needed. Here we provide the overall study protocol, including approaches for subject recruitment, strategies for imaging typically developing children 0–5 years of age without sedation, imaging protocol and optimization, a description of the battery of behavioral assessments, and QA/QC procedures. Combining HCP inspired neuroimaging data with well-established behavioral assessments during this time period will yield an invaluable resource for the scientific community.

YNIMG Journal 2019 Journal Article

Topological correction of infant white matter surfaces using anatomically constrained convolutional neural network

• Liang Sun
• Daoqiang Zhang
• Chunfeng Lian
• Li Wang
• Zhengwang Wu
• WEI SHAO
• Weili Lin
• Dinggang Shen

Reconstruction of accurate cortical surfaces without topological errors (i. e. , handles and holes) from infant brain MR images is very important in early brain development studies. However, infant brain MR images typically suffer extremely low tissue contrast and dynamic imaging appearance patterns. Thus, it is inevitable to have large amounts of topological errors in the segmented infant brain tissue images, which lead to inaccurately reconstructed cortical surfaces with topological errors. To address this issue, inspired by recent advances in deep learning, we propose an anatomically constrained network for topological correction on infant cortical surfaces. Specifically, in our method, we first locate regions of potential topological defects by leveraging a topology-preserving level set method. Then, we propose an anatomically constrained network to correct those candidate voxels in the located regions. Since infant cortical surfaces often contain large and complex handles or holes, it is difficult to completely correct all errors using one-shot correction. Therefore, we further enroll these two steps into an iterative framework to gradually correct large topological errors. To the best of our knowledge, this is the first work to introduce deep learning approach for topological correction of infant cortical surfaces. We compare our method with the state-of-the-art methods on both simulated topological errors and real topological errors in human infant brain MR images. Moreover, we also validate our method on the infant brain MR images of macaques. All experimental results show the superior performance of the proposed method.

YNIMG Journal 2017 Journal Article

Joint prediction of longitudinal development of cortical surfaces and white matter fibers from neonatal MRI

• Islem Rekik
• Gang Li
• Pew-Thian Yap
• Geng Chen
• Weili Lin
• Dinggang Shen

The human brain can be modeled as multiple interrelated shapes (or a multishape), each for characterizing one aspect of the brain, such as the cortex and white matter pathways. Predicting the developing multishape is a very challenging task due to the contrasting nature of the developmental trajectories of the constituent shapes: smooth for the cortical surface and non-smooth for white matter tracts due to changes such as bifurcation. We recently addressed this problem and proposed an approach for predicting the multishape developmental spatiotemporal trajectories of infant brains based only on neonatal MRI data using a set of geometric, dynamic, and fiber-to-surface connectivity features. In this paper, we propose two key innovations to further improve the prediction of multishape evolution. First, for a more accurate cortical surface prediction, instead of simply relying on one neonatal atlas to guide the prediction of the multishape, we propose to use multiple neonatal atlases to build a spatially heterogeneous atlas using the multidirectional varifold representation. This individualizes the atlas by locally maximizing its similarity to the testing baseline cortical shape for each cortical region, thereby better representing the baseline testing cortical surface, which founds the multishape prediction process. Second, for temporally consistent fiber prediction, we propose to reliably estimate spatiotemporal connectivity features using low-rank tensor completion, thereby capturing the variability and richness of the temporal development of fibers. Experimental results confirm that the proposed variants significantly improve the prediction performance of our original multishape prediction framework for both cortical surfaces and fiber tracts shape at 3, 6, and 9 months of age. Our pioneering model will pave the way for learning how to predict the evolution of anatomical shapes with abnormal changes. Ultimately, devising accurate shape evolution prediction models that can help quantify and predict the severity of a brain disorder as it progresses will be of great aid in individualized treatment planning.

YNIMG Journal 2016 Journal Article

Multidirectional and Topography-based Dynamic-scale Varifold Representations with Application to Matching Developing Cortical Surfaces

• Islem Rekik
• Gang Li
• Weili Lin
• Dinggang Shen

The human cerebral cortex is marked by great complexity as well as substantial dynamic changes during early postnatal development. To obtain a fairly comprehensive picture of its age-induced and/or disorder-related cortical changes, one needs to match cortical surfaces to one another, while maximizing their anatomical alignment. Methods that geodesically shoot surfaces into one another as currents (a distribution of oriented normals) and varifolds (a distribution of non-oriented normals) provide an elegant Riemannian framework for generic surface matching and reliable statistical analysis. However, both conventional current and varifold matching methods have two key limitations. First, they only use the normals of the surface to measure its geometry and guide the warping process, which overlooks the importance of the orientations of the inherently convoluted cortical sulcal and gyral folds. Second, the ‘conversion’ of a surface into a current or a varifold operates at a fixed scale under which geometric surface details will be neglected, which ignores the dynamic scales of cortical foldings. To overcome these limitations and improve varifold-based cortical surface registration, we propose two different strategies. The first strategy decomposes each cortical surface into its normal and tangent varifold representations, by integrating principal curvature direction field into the varifold matching framework, thus providing rich information of the orientation of cortical folding and better characterization of the complex cortical geometry. The second strategy explores the informative cortical geometric features to perform a dynamic-scale measurement of the cortical surface that depends on the local surface topography (e. g. , principal curvature), thereby we introduce the concept of a topography-based dynamic-scale varifold. We tested the proposed varifold variants for registering 12 pairs of dynamically developing cortical surfaces from 0 to 6 months of age. Both variants improved the matching accuracy in terms of closeness to the target surface and the goodness of alignment with regional anatomical boundaries, when compared with three state-of-the-art methods: (1) diffeomorphic spectral matching, (2) conventional current-based surface matching, and (3) conventional varifold-based surface matching.

YNIMG Journal 2015 Journal Article

LINKS: Learning-based multi-source IntegratioN frameworK for Segmentation of infant brain images

• Li Wang
• Yaozong Gao
• Feng Shi
• Gang Li
• John H. Gilmore
• Weili Lin
• Dinggang Shen

Segmentation of infant brain MR images is challenging due to insufficient image quality, severe partial volume effect, and ongoing maturation and myelination processes. In the first year of life, the image contrast between white and gray matters of the infant brain undergoes dramatic changes. In particular, the image contrast is inverted around 6–8months of age, and the white and gray matter tissues are isointense in both T1- and T2-weighted MR images and thus exhibit the extremely low tissue contrast, which poses significant challenges for automated segmentation. Most previous studies used multi-atlas label fusion strategy, which has the limitation of equally treating the different available image modalities and is often computationally expensive. To cope with these limitations, in this paper, we propose a novel learning-based multi-source integration framework for segmentation of infant brain images. Specifically, we employ the random forest technique to effectively integrate features from multi-source images together for tissue segmentation. Here, the multi-source images include initially only the multi-modality (T1, T2 and FA) images and later also the iteratively estimated and refined tissue probability maps of gray matter, white matter, and cerebrospinal fluid. Experimental results on 119 infants show that the proposed method achieves better performance than other state-of-the-art automated segmentation methods. Further validation was performed on the MICCAI grand challenge and the proposed method was ranked top among all competing methods. Moreover, to alleviate the possible anatomical errors, our method can also be combined with an anatomically-constrained multi-atlas labeling approach for further improving the segmentation accuracy.

YNIMG Journal 2014 Journal Article

Integration of sparse multi-modality representation and anatomical constraint for isointense infant brain MR image segmentation

• Li Wang
• Feng Shi
• Yaozong Gao
• Gang Li
• John H. Gilmore
• Weili Lin
• Dinggang Shen

Segmentation of infant brain MR images is challenging due to poor spatial resolution, severe partial volume effect, and the ongoing maturation and myelination processes. During the first year of life, the brain image contrast between white and gray matters undergoes dramatic changes. In particular, the image contrast inverses around 6–8months of age, where the white and gray matter tissues are isointense in T1 and T2 weighted images and hence exhibit the extremely low tissue contrast, posing significant challenges for automated segmentation. In this paper, we propose a general framework that adopts sparse representation to fuse the multi-modality image information and further incorporate the anatomical constraints for brain tissue segmentation. Specifically, we first derive an initial segmentation from a library of aligned images with ground-truth segmentations by using sparse representation in a patch-based fashion for the multi-modality T1, T2 and FA images. The segmentation result is further iteratively refined by integration of the anatomical constraint. The proposed method was evaluated on 22 infant brain MR images acquired at around 6months of age by using a leave-one-out cross-validation, as well as other 10 unseen testing subjects. Our method achieved a high accuracy for the Dice ratios that measure the volume overlap between automated and manual segmentations, i. e. , 0. 889±0. 008 for white matter and 0. 870±0. 006 for gray matter.

YNIMG Journal 2014 Journal Article

Measuring the dynamic longitudinal cortex development in infants by reconstruction of temporally consistent cortical surfaces

• Gang Li
• Jingxin Nie
• Li Wang
• Feng Shi
• John H. Gilmore
• Weili Lin
• Dinggang Shen

Quantitative measurement of the dynamic longitudinal cortex development during early postnatal stages is of great importance to understand the early cortical structural and functional development. Conventional methods usually reconstruct the cortical surfaces of longitudinal images from the same subject independently, which often generate longitudinally-inconsistent cortical surfaces and thus lead to inaccurate measurement of cortical changes, especially for vertex-wise mapping of cortical development. This paper aims to address this problem by presenting a method to reconstruct temporally-consistent cortical surfaces from longitudinal infant brain MR images, for accurate and consistent measurement of the dynamic cortex development in infants. Specifically, the longitudinal development of the inner cortical surface is first modeled by a deformable growth sheet with elasto-plasticity property to establish longitudinally smooth correspondences of the inner cortical surfaces. Then, the modeled longitudinal inner cortical surfaces are jointly deformed to locate both inner and outer cortical surfaces with a spatial–temporal deformable surface method. The method has been applied to 13 healthy infants, each with 6 serial MR scans acquired at 2weeks, 3months, 6months, 9months, 12months and 18months of age. Experimental results showed that our method with the incorporated longitudinal constraints can reconstruct the longitudinally-dynamic cortical surfaces from serial infant MR images more consistently and accurately than the previously published methods. By using our method, for the first time, we can characterize the vertex-wise longitudinal cortical thickness development trajectory at multiple time points in the first 18months of life. Specifically, we found the highly age-related and regionally-heterogeneous developmental trajectories of the cortical thickness during this period, with the cortical thickness increased most from 3 to 6months (16. 2%) and least from 9 to 12months (less than 0. 1%). Specifically, the central sulcus only underwent significant increase of cortical thickness from 6 to 9months and the occipital cortex underwent significant increase from 0 to 9months, while the frontal, temporal and parietal cortices grew continuously in this first 18months of life. The adult-like spatial patterns of cortical thickness were generally present at 18months of age. These results provided detailed insights into the dynamic trajectory of the cortical thickness development in infants.

IROS Conference 2014 Conference Paper

Position control of a robot end-effector based on synthetic aperture wireless localization

• Albert Marschall
• Thorsten Voigt
• Gang Li
• Ulrich Konigorski
• Martin Vossiek

The implementation of lightweight robot concepts requires novel measurement and control strategies to deal with the flexibility of the robot arm. To the authors' best knowledge this paper introduces for the first time an innovative closed-loop concept to measure and control the absolute position of the tool center point (TCP) based on a synthetic aperture wireless localization approach. A radio-frequency identification (RFID)-like backscatter transponder is attached to the TCP, and several radar base stations measure the respective roundtrip time-of-flight and phase of the backscattered transponder signals. Inverse radar apertures are synthesized based on a small movement of the TCP. The small TCP trajectory is tracked with simple relative sensors. A holographic synthetic aperture reconstruction algorithm then determines the absolute TCP position - i. e. , the trajectory coordinates in the world coordinate system. This absolute positional information is used to observe the control system and identify model errors, which are minimized in a closed-loop procedure. This iteratively improves both the system model and control quality as well as the precision of the synthetic aperture wireless localization. System simulations and first test results prove that the novel concept is suited to measure and control the position of an end-effector with mm-precision, even if the initial system model is partly erroneous.

YNIMG Journal 2014 Journal Article

Segmentation of neonatal brain MR images using patch-driven level sets

• Li Wang
• Feng Shi
• Gang Li
• Yaozong Gao
• Weili Lin
• John H. Gilmore
• Dinggang Shen

The segmentation of neonatal brain MR image into white matter (WM), gray matter (GM), and cerebrospinal fluid (CSF), is challenging due to the low spatial resolution, severe partial volume effect, high image noise, and dynamic myelination and maturation processes. Atlas-based methods have been widely used for guiding neonatal brain segmentation. Existing brain atlases were generally constructed by equally averaging all the aligned template images from a population. However, such population-based atlases might not be representative of a testing subject in the regions with high inter-subject variability and thus often lead to a low capability in guiding segmentation in those regions. Recently, patch-based sparse representation techniques have been proposed to effectively select the most relevant elements from a large group of candidates, which can be used to generate a subject-specific representation with rich local anatomical details for guiding the segmentation. Accordingly, in this paper, we propose a novel patch-driven level set method for the segmentation of neonatal brain MR images by taking advantage of sparse representation techniques. Specifically, we first build a subject-specific atlas from a library of aligned, manually segmented images by using sparse representation in a patch-based fashion. Then, the spatial consistency in the probability maps from the subject-specific atlas is further enforced by considering the similarities of a patch with its neighboring patches. Finally, the probability maps are integrated into a coupled level set framework for more accurate segmentation. The proposed method has been extensively evaluated on 20 training subjects using leave-one-out cross validation, and also on 132 additional testing subjects. Our method achieved a high accuracy of 0. 919±0. 008 for white matter and 0. 901±0. 005 for gray matter, respectively, measured by Dice ratio for the overlap between the automated and manual segmentations in the cortical region.

YNIMG Journal 2014 Journal Article

Spatial distribution and longitudinal development of deep cortical sulcal landmarks in infants

• Yu Meng
• Gang Li
• Weili Lin
• John H. Gilmore
• Dinggang Shen

Sulcal pits, the locally deepest points in sulci of the highly convoluted and variable cerebral cortex, are found to be spatially consistent across human adult individuals. It is suggested that sulcal pits are genetically controlled and have close relationships with functional areas. To date, the existing imaging studies of sulcal pits are mainly focused on adult brains, yet little is known about the spatial distribution and temporal development of sulcal pits in the first 2years of life, which is the most dynamic and critical period of postnatal brain development. Studying sulcal pits during this period would greatly enrich our limited understandings of the origins and developmental trajectories of sulcal pits, and would also provide important insights into many neurodevelopmental disorders associated with abnormal cortical foldings. In this paper, by using surface-based morphometry, for the first time, we systemically investigated the spatial distribution and temporal development of sulcal pits in major cortical sulci from 73 healthy infants, each with three longitudinal 3T MR scans at term birth, 1year, and 2years of age. Our results suggest that the spatially consistent distributions of sulcal pits in major sulci across individuals have already existed at term birth and this spatial distribution pattern keeps relatively stable in the first 2years of life, despite that the cerebral cortex expands dramatically and the sulcal depth increases considerably during this period. Specially, the depth of sulcal pits increases regionally heterogeneously, with more rapid growth in the high-order association cortex, including the prefrontal and temporal cortices, than the sensorimotor cortex in the first 2years of life. Meanwhile, our results also suggest that there exist hemispheric asymmetries of the spatial distributions of sulcal pits in several cortical regions, such as the central, superior temporal and postcentral sulci, consistently from birth to 2years of age, which likely has close relationships with the lateralization of brain functions of these regions. This study provides detailed insights into the spatial distribution and temporal development of deep sulcal landmarks in infants.

YNIMG Journal 2013 Journal Article

Development of cortical anatomical properties from early childhood to early adulthood

• Jingxin Nie
• Gang Li
• Dinggang Shen

Human brain matures in temporal and regional heterogeneity, with some areas matured at early adulthood. In this study, the relationship of cortical structural developments between different cortical sheet regions is systematically analyzed using interregional correlation coefficient and network methods. Specifically, 951 longitudinal T1 brain MR images from 445 healthy subjects with ages from 3 to 20years old are used. The result shows that the development of cortex reaches a turning point at around 7years of age: a) the cortical thickness reaches its highest value and also the cortical folding becomes stable at this age; b) both global and local efficiencies of anatomical correlation networks reach the lowest and highest values at this age, respectively; and c) the change of anatomical correlation networks reach the highest level at this age, and the convergence of different anatomical correlation networks starts to decrease from this age. These results might inspire more studies on why there exists a turning point at this age from different viewpoints. For example, is there any change of synaptic pruning, or is it related to the starting of school life? And how can we benefit from this in the real life?

YNIMG Journal 2012 Journal Article

Consistent reconstruction of cortical surfaces from longitudinal brain MR images

• Gang Li
• Jingxin Nie
• Guorong Wu
• Yaping Wang
• Dinggang Shen

Accurate and consistent reconstruction of cortical surfaces from longitudinal human brain MR images is of great importance in studying longitudinal subtle change of the cerebral cortex. This paper presents a novel deformable surface method for consistent and accurate reconstruction of inner, central and outer cortical surfaces from longitudinal brain MR images. Specifically, the cortical surfaces of the group-mean image of all aligned longitudinal images of the same subject are first reconstructed by a deformable surface method, which is driven by a force derived from the Laplace's equation. And then the longitudinal cortical surfaces are consistently reconstructed by jointly deforming the cortical surfaces of the group-mean image to all longitudinal images. The proposed method has been successfully applied to two sets of longitudinal human brain MR images. Both qualitative and quantitative experimental results demonstrate the accuracy and consistency of the proposed method. Furthermore, the reconstructed longitudinal cortical surfaces are used to measure the longitudinal changes of cortical thickness in both normal and diseased groups, where the overall decline trend of cortical thickness has been clearly observed. Meanwhile, the longitudinal cortical thickness also shows its potential in distinguishing different clinical groups.

ICRA Conference 2012 Conference Paper

Robot semantic mapping through wearable sensor-based human activity recognition

• Gang Li
• Chun Zhu
• Jianhao Du
• Qi Cheng 0002
• Weihua Sheng
• Heping Chen

Semantic information can help both humans and robots to understand their environments better. In order to obtain semantic information efficiently and link it to a metric map, we present a semantic mapping approach through human activity recognition in an indoor human-robot coexisting environment. An intelligent mobile robot platform can create a 2D metric map, while human activity can be recognized using motion data from wearable motion sensors mounted on a human subject. Combined with pre-learned models of activity-to-furniture type association and robot pose estimates, the robot can determine the distribution of the furniture types on the 2D metric map. Simulations and real world experiments demonstrate that the proposed method is able to create a reliable metric map with accurate semantic information.

YNIMG Journal 2011 Journal Article

Consistent sulcal parcellation of longitudinal cortical surfaces

• Gang Li
• Dinggang Shen

Automated accurate and consistent sulcal parcellation of longitudinal cortical surfaces is of great importance in studying longitudinal morphological and functional changes of human brains, since longitudinal cortical changes are normally very subtle, especially in aging brains. However, applying the existing methods (which were typically developed for cortical sulcal parcellation of a single cortical surface) independently to longitudinal cortical surfaces might generate longitudinally-inconsistent results. To overcome this limitation, this paper presents a novel energy function based method for accurate and consistent sulcal parcellation of longitudinal cortical surfaces. Specifically, both spatial and temporal smoothness are imposed in the energy function to obtain consistent longitudinal sulcal parcellation results. The energy function is efficiently minimized by a graph cut method. The proposed method has been successfully applied to sulcal parcellation of both real and simulated longitudinal inner cortical surfaces of human brain MR images. Both qualitative and quantitative evaluation results demonstrate the validity of the proposed method.

YNIMG Journal 2010 Journal Article

Gyral folding pattern analysis via surface profiling

• Kaiming Li
• Lei Guo
• Gang Li
• Jingxin Nie
• Carlos Faraco
• Guangbin Cui
• Qun Zhao
• L. Stephen Miller

Folding is an essential shape characteristic of the human cerebral cortex. Descriptors of cortical folding patterns have been studied for decades. However, many previous studies are either based on local shape descriptors such as curvature, or based on global descriptors such as gyrification index or spherical wavelets. This paper proposes a gyrus-scale folding pattern analysis technique via cortical surface profiling. Firstly, we sample the cortical surface into 2D profiles and model them using a power function. This step provides both the flexibility of representing arbitrary shape by profiling and the compactness of representing shape by parametric modeling. Secondly, based on the estimated model parameters, we extract affine-invariant features on the cortical surface, and apply the affinity propagation clustering algorithm to parcellate the cortex into cortical regions with strict hierarchy and smooth transitions among them. Finally, a second-round surface profiling is performed on the parcellated cortical surface, and the number of hinges is detected to describe the gyral folding pattern. We have applied the surface profiling method to two normal brain datasets and a schizophrenia patient dataset. The experimental results demonstrate that the proposed method can accurately classify human gyri into 2-hinge, 3-hinge and 4-hinge patterns. The distribution of these folding patterns on brain lobes and the relationship between fiber density and gyral folding patterns are further investigated. Results from the schizophrenia dataset are consistent with commonly found abnormality in former studies by others, which demonstrates the potential clinical applications of the proposed technique.

YNIMG Journal 2009 Journal Article

Automatic cortical sulcal parcellation based on surface principal direction flow field tracking

• Gang Li
• Lei Guo
• Jingxin Nie
• Tianming Liu

The human cerebral cortex is a highly convoluted structure composed of sulci and gyri, corresponding to the valleys and ridges of the cortical surface respectively. Automatic parcellation of the cortical surface into sulcal regions is of great importance in structural and functional mapping of the human brain. In this paper, a novel method is proposed for automatic cortical sulcal parcellation based on the geometric characteristics of cortical surface including its principal curvatures and principal directions. This method is composed of two major steps: 1) employing the hidden Markov random field model (HMRF) and the expectation maximization (EM) algorithm on the maximum principal curvatures of the cortical surface for sulcal region segmentation, and 2) using a principal direction flow field tracking method on the cortical surface for sulcal basin segmentation. The flow field is obtained by diffusing the principal direction field on the cortical surface mesh. A unique feature of this method is that the automatic sulcal parcellation process is quite robust and efficient, and is independent of any external guidance such as atlas-based warping. The method has been successfully applied to the inner cortical surfaces of twelve healthy human brain MR images. Both quantitative and qualitative evaluation results demonstrate the validity and efficiency of the proposed method.

SODA Conference 1999 Conference Paper

The Advantages of Forward Thinking in Generating Rooted and Free Trees

• Gang Li
• Frank Ruskey