Dinggang Shen

JBHI Journal 2026 Journal Article

MGTP: Multi-Granularity Textual Prompts for Low-Dose Brain PET Image Denoising via Adversarial Diffusion Model

• Jiaqi Cui
• Xinyi Zeng
• Pinxian Zeng
• Bo Liu
• Xi Wu
• Deng Xiong
• Jiliu Zhou
• Yan Wang

Positron emission tomography (PET) is an advanced nuclear imaging technique and has been widely applied in clinic. However, radiation risks associated with standard-dose PET imaging raise health concerns, whereas the quality of low-dose PET images fails to meet clinical requirements. To reduce the tracer dose while maintaining image quality, it is of great interest to estimate high-quality PET images from low-dose images. However, existing low-dose PET image denoising methods primarily focus on image data, overlooking crucial information in non-image textual data such as patients’ clinical tabular and textual descriptions of general image quality. This neglect can lead to subpar denoising quality with inaccurate contexts and poor details. To address these problems, in this paper, we propose Multi-Granularity Textual Prompts, namely MGTP, to denoise low-dose PET images via an adversarial diffusion model. Different from prior methods that rely solely on image conditioning, our MGTP innovatively introduces textual prompts spanning diverse granularities to capture both high-level semantic-related contexts and low-level degradation-related details. To harmonize multi-granularity textual prompts with low-dose PET images, we design a Cross-Modality Selective Conditioning (CMSC) module, which prioritizes semantic- and detail-relevant information while eliminating irrelevant components. The resulting features are fed into diffusion model as conditions, enforcing a more controlled diffusion process. In addition, we develop a Masked Prompt Reconstruction Network (MPR-Net) to enhance the preservation of semantics and details in denoised images, mitigating distortions brought by the random noise in the diffusion process. Experiments on clinical PET data show that our method achieves the state-of-the-art performance.

JBHI Journal 2026 Journal Article

Pancreas Segmentation With Multi-Phase Feature Aggregation and Modality Adaptive Transformer

• Lulu Tan
• Wenda Sheng
• Jiadong Zhang
• Wenbin Zou
• DengQiang Jia
• Qianqian Chen
• Jing Sheng
• Yangyang Qian

Automatic pancreas segmentation can facilitate diagnosis and treatment of pancreatic diseases. The combination of non-contrast, arterial, and venous phases of CT imaging can enhance differentiation of the pancreas from its surrounding structures. However, existing multimodal methods, which try to integrate the multimodal information in computer-aided pancreas segmentation, often overlook the inter-modal relationships and have a limited capability for information fusion. In this paper, we propose a multi-phase pancreas segmentation method for incorporating Feature Aggregation Module (FAM) and Modality Adaptive Transformer (MAT). Specifically, we use the venous phase as the primary modality, while the non-contrast and arterial phases serve as supplementary modalities, based on clinical prior knowledge. Our FAM integrates spatial information from the primary and supplementary modalities, while our MAT adaptively enhances feature representation and establishes long-range dependencies among modalities. Our method outperforms state-of-the-art techniques on a large scale dataset. Based on the segmented pancreas region, We further perform a downstream task focused on pancreatic volume calculation. The prediction accuracy is on par with manual segmentation, demonstrating effectiveness and potential application of our proposed method.

JBHI Journal 2025 Journal Article

A Modality-Flexible Framework for Alzheimer's Disease Diagnosis Following Clinical Routine

• Yuanwang Zhang
• Kaicong Sun
• Yuxiao Liu
• Fang Xie
• Qihao Guo
• Dinggang Shen

Dementia has high incidence among the elderly, and Alzheimer's disease (AD) is the most common dementia. The procedure of AD diagnosis in clinics usually follows a standard routine consisting of different phases, from acquiring non-imaging tabular data in the screening phase to MR imaging and ultimately to PET imaging. Most of the existing AD diagnosis studies are dedicated to a specific phase using either single or multi-modal data. In this paper, we introduce a modality-flexible classification framework, which is applicable for different AD diagnosis phases following the clinical routine. Specifically, our framework consists of three branches corresponding to three diagnosis phases: 1) a tabular branch using only tabular data for screening phase, 2) an MRI branch using both MRI and tabular data for uncertain cases in screening phase, and 3) ultimately a PET branch for the challenging cases using all the modalities including PET, MRI, and tabular data. To achieve effective fusion of imaging and non-imaging modalities, we introduce an image-tabular transformer block to adaptively scale and shift the image and tabular features according to modality importance determined by the network. The proposed framework is extensively validated on four cohorts containing 6495 subjects. Experiments demonstrate that our framework achieves superior diagnostic performance than the other representative methods across various AD diagnosis tasks, and shows promising performance for all the diagnosis phases, which exhibits great potential for clinical application.

JBHI Journal 2025 Journal Article

Aleatoric-Uncertainty-Aware Maximum Intensity Projection-Based GAN for 7T-Like Generation From 3T TOF-MRA

• Wei Tang
• Yuxiang Dai
• Boyu Zhang
• Zhang Shi
• Ying-Hua Chu
• Peixian Zhuang
• Dinggang Shen
• Chengyan Wang

Time-of-flight magnetic resonance angiography (TOF-MRA) is a prevalent vascular imaging technique for assessing cerebrovascular diseases. Compared to routine 3T TOF-MRA, 7T TOF-MRA provides vascular structures with a higher signal-to-noise ratio (SNR) and better vessel contrast, revealing greater vascular details. However, the inaccessibility of 7T scanners and specific physiological and technical concerns limit its clinical application. Therefore, we aimed to generate high-quality 7T-like TOF-MRA from 3T TOF-MRA. Considering the spatial sparsity of vessel signals, the visibility discrepancy of distal and small vessels between 3T and 7T images, and the subtle spatial misalignment between paired data, we proposed a novel aleatoric-uncertainty-aware maximum intensity projection-based generative adversarial network (AU-MIPGAN). In our method, we employed a knowledge distillation (KD) framework to incorporate multi-directional MIP information into the 3T-to-7T learning process to strengthen the learning of vessels and provide three-dimensional (3D) vascular morphological knowledge for the student model, facilitating accurate generation of vascular structures. Furthermore, we exploited AU modeling to compensate for the spatial misalignment between paired 3T and 7T images during the training procedure, which helped the model concentrate more on learning the intrinsic gap between 3T and 7T images. Qualitative and quantitative results demonstrated that the proposed AU-MIPGAN can achieve promising performance for 7T-like TOF-MRA generation.

JBHI Journal 2025 Journal Article

Image-and-Label Conditioning Latent Diffusion Model: Synthesizing A$\beta$-PET From MRI for Detecting Amyloid Status

• Zaixin Ou
• Yongsheng Pan
• Fang Xie
• Qihao Guo
• Dinggang Shen

Deposition of $\beta$ -amyloid (A $\beta$ ), which is generally observed by A $\beta$ -PET, is an important biomarker to evaluate subjects with early-onset dementia. However, acquisition of A $\beta$ -PET usually suffers from high expense and radiation hazards, making A $\beta$ -PET not commonly used as MRI. As A $\beta$ -PET scans are only used to determine whether A $\beta$ deposition is positive or not, it is highly valuable to capture the underlying relationship between A $\beta$ deposition and other neuroimages (i. e. , MRI) and detect amyloid status based on other neuroimages to reduce necessity of acquiring A $\beta$ -PET. To this end, we propose an image-and-label conditioning latent diffusion model (IL-CLDM) to synthesize A $\beta$ -PET scans from MRI scans by enhancing critical shared information to finally achieve MRI-based A $\beta$ classification. Specifically, two conditioning modules are introduced to enable IL-CLDM to implicitly learn joint image synthesis and diagnosis: 1) an image conditioning module, to extract meaningful features from source MRI scans to provide structural information, and 2) a label conditioning module, to guide the alignment of generated scans to the diagnosed label. Experiments on a clinical dataset of 510 subjects demonstrate that our proposed IL-CLDM achieves image quality superior to five widely used models, and our synthesized A $\beta$ -PET scans (by IL-CLDM) can significantly help classification of A $\beta$ as positive or negative.

JBHI Journal 2025 Journal Article

Structure-Aware Brain Tissue Segmentation for Isointense Infant MRI Data Using Multi-Phase Multi-Scale Assistance Network

• Jiameng Liu
• Feihong Liu
• Dong Nie
• Yuning Gu
• Yuhang Sun
• Dinggang Shen

Accurate and automatic brain tissue segmentation is crucial for tracking brain development and diagnosing brain disorders. However, due to inherently ongoing myelination and maturation during the first postnatal year, the intensity distributions of gray matter and white matter in the infant brain MRI at the age of around 6 months old ( a. k. a. isointense phase) are highly overlapped, which makes tissue segmentation very challenging, even for experts. To address this issue, in this study, we propose a multi-phase multi-scale assistance segmentation framework, which comprises a structure-preserved generative adversarial network (SPGAN) and a multi-phase multi-scale assisted segmentation network (M $^{2}$ ASN). SPGAN bi-directionally synthesizes isointense and adult-like data. The synthetic isointense data essentially augment the training dataset, combined with high-quality annotations transferred from its adult-like counterpart. By contrast, the synthetic adult-like data offers clear tissue structures and is concatenated with isointense data to serve as the input of M $^{2}$ ASN. In particular, M $^{2}$ ASN is designed with two-branch networks, which simultaneously segment tissues with two phases (isointense and adult-like) and two scales by also preserving their correspondences. We further propose a boundary refinement module to extract maximum gradients from local feature maps to indicate tissue boundaries, prompting M $^{2}$ ASN to focus more on boundaries where segmentation errors are prone to occur. Extensive experiments on the National Database for Autism Research and Baby Connectome Project datasets quantitatively and qualitatively demonstrate the superiority of our proposed framework compared with seven state-of-the-art methods.

JBHI Journal 2025 Journal Article

Super-resolution Reconstruction of Fetal Brain MRI with Multi-view Interpolation Weight Learning

• Shijie Huang
• DengQiang Jia
• Kai Zhang
• Lingnan Kong
• Fangmei Zhu
• Zhongxiang Ding
• Geng Chen
• Dinggang Shen

Super-resolution reconstruction (SRR) of isotropic fetal brain MR images is critical for prenatal ex aminations but is hindered by fetal motion and misalignment of thick-slice scans. To address these challenges comprehensively, we introduce an innovative deep learning model, namely 3D-WISE, a 3D Weighted Interpolation for Super-resolution Estimation of fetal brain MRI. The model generates high-quality isotropic fetal brain MR images by learning the interpolation weights to correct misalignments between slices and volumes. These misalignments are estimated by extracting deep features from multiple motion corrupted stacks. Specifically, 3D-WISE incorporates two key components: (1) a weight learning module for multi view interpolation and (2) a feature extraction module guided by multi-type attention mechanisms. The weight learning module first maps motion-corrupted thick-slice stacks into latent feature spaces. The resulting features are then fed to an implicit decoding block to estimate interpolation weights of the surrounding points for a given coordinate. We further enhance our approach by incorporating convolutional block attention and atlas-induced cross-attention mechanisms. Extensive experiments on two benchmark datasets show that our 3D-WISE achieves remarkably improved performance compared to the widely adoptedregistration-reconstruction framework. We also ex tend the experiments on anatomical structure reconstruction and achieve promising results, highlighting the significant potential of our 3D-WISE for fetal brain MR images SRR in clinical settings. Our code is available at https://github.com/sj-huang/3D-WISE.

JBHI Journal 2025 Journal Article

Unified Model for Children's Brain Image Segmentation With Co-Registration Framework Guided by Longitudinal MRI

• Lin Teng
• Yichu He
• Zehong Cao
• Rui Hua
• Ye Han
• Qianjin Feng
• Feng Shi
• Dinggang Shen

Accurate segmentation of brain structures is crucial for analyzing longitudinal changes in children's brains. However, existing methods are mostly based on models established at a single time-point due to difficulty in obtaining annotated data and dynamic variation of tissue intensity. The main problem with such approaches is that, when conducting longitudinal analysis, images from different time points are segmented by different models, leading to significant variation in estimating development trends. In this paper, we propose a novel unified model with co-registration framework to segment children's brain images covering neonates to preschoolers, which is formulated as two stages. First, to overcome the shortage of annotated data, we propose building gold-standard segmentation with co-registration framework guided by longitudinal data. Second, we construct a unified segmentation model tailored to brain images at 0-6 years old through the introduction of a convolutional network (named SE-VB-Net), which combines our previously proposed VB-Net with Squeeze-and-Excitation (SE) block. Moreover, different from existing methods that only require both T1- and T2-weighted MR images as inputs, our designed model also allows a single T1-weighted MR image as input. The proposed method is evaluated on the main dataset (320 longitudinal subjects with average 2 time-points) and two external datasets (10 cases with 6-month-old and 40 cases with 20–45 weeks, respectively). Results demonstrate that our proposed method achieves a high performance (>92%), even over a single time-point. This means that it is suitable for brain image analysis with large appearance variation, and largely broadens the application scenarios.

IJCAI Conference 2024 Conference Paper

A Prior-information-guided Residual Diffusion Model for Multi-modal PET Synthesis from MRI

• Zaixin Ou
• Caiwen Jiang
• Yongsheng Pan
• Yuanwang Zhang
• Zhiming Cui
• Dinggang Shen

Alzheimer's disease (AD) leads to abnormalities in various biomarkers (i. e. , amyloid-β and tau proteins), which makes PET imaging (which can detect these biomarkers) essential in AD diagnosis. However, the high radiation risk of PET imaging limits its scanning number within a short period, presenting challenges to the joint multi-biomarker diagnosis of AD. In this paper, we propose a novel unified model to simultaneously synthesize multi-modal PET images from MRI, to achieve low-cost and time-efficient joint multi-biomarker diagnosis of AD. Specifically, we incorporate residual learning into the diffusion model to emphasize inter-domain differences between PET and MRI, thereby forcing each modality to maximally reconstruct its modality-specific details. Furthermore, we leverage prior information, such as age and gender, to guide the diffusion model in synthesizing PET images with semantic consistency, enhancing their diagnostic value. Additionally, we develop an intra-domain difference loss to ensure that the intra-domain differences among synthesized PET images closely match those among real PET images, promoting more accurate synthesis, especially at the modality-specific information. Extensive experiments conducted on the ADNI dataset demonstrate that our method achieves superior performance both quantitatively and qualitatively compared to the state-of-the-art methods. All codes for this study have been uploaded to GitHub (https: //github. com/Ouzaixin/ResDM).

JBHI Journal 2024 Journal Article

A Unified Multi-Modality Fusion Framework for Deep Spatio-Spectral-Temporal Feature Learning in Resting-State fMRI Denoising

• Minjoo Lim
• Keun-Soo Heo
• Jun-Mo Kim
• Bogyeong Kang
• Weili Lin
• Han Zhang
• Dinggang Shen
• Tae-Eui Kam

Resting-state functional magnetic resonance imaging (rs-fMRI) is a commonly used functional neuroimaging technique to investigate the functional brain networks. However, rs-fMRI data are often contaminated with noise and artifacts that adversely affect the results of rs-fMRI studies. Several machine/deep learning methods have achieved impressive performance to automatically regress the noise-related components decomposed from rs-fMRI data, which are expressed as the pairs of a spatial map and its associated time series. However, most of the previous methods individually analyze each modality of the noise-related components and simply aggregate the decision-level information (or knowledge) extracted from each modality to make a final decision. Moreover, these approaches consider only the limited modalities making it difficult to explore class-discriminative spectral information of noise-related components. To overcome these limitations, we propose a unified deep attentive spatio-spectral-temporal feature fusion framework. We first adopt a learnable wavelet transform module at the input-level of the framework to elaborately explore the spectral information in subsequent processes. We then construct a feature-level multi-modality fusion module to efficiently exchange the information from multi-modality inputs in the feature space. Finally, we design confidence-based voting strategies for decision-level fusion at the end of the framework to make a robust final decision. In our experiments, the proposed method achieved remarkable performance for noise-related component detection on various rs-fMRI datasets.

JBHI Journal 2024 Journal Article

Attention-Based MultiOffset Deep Learning Reconstruction of Chemical Exchange Saturation Transfer (AMO-CEST) MRI

• Zhikai Yang
• Dinggang Shen
• Kannie W. Y. Chan
• Jianpan Huang

One challenge of chemical exchange saturation transfer (CEST) magnetic resonance imaging (MRI) is the long scan time due to multiple acquisitions of images at different saturation frequency offsets. k-space under-sampling strategy is commonly used to accelerate MRI acquisition, while this could introduce artifacts and reduce signal-to-noise ratio (SNR). To accelerate CEST-MRI acquisition while maintaining suitable image quality, we proposed an attention-based multioffset deep learning reconstruction network (AMO-CEST) with a multiple radial k-space sampling strategy for CEST-MRI. The AMO-CEST also contains dilated convolution to enlarge the receptive field and data consistency module to preserve the sampled k-space data. We evaluated the proposed method on a mouse brain dataset containing 5760 CEST images acquired at a pre-clinical 3 T MRI scanner. Quantitative results demonstrated that AMO-CEST showed obvious improvement over zero-filling method with a PSNR enhancement of 11 dB, a SSIM enhancement of 0. 15, and a NMSE decrease of $\text{4. 37}\times \text{10}^{-\text{2}}$ in three acquisition orientations. Compared with other deep learning-based models, AMO-CEST showed visual and quantitative improvements in images from three different orientations. We also extracted molecular contrast maps, including the amide proton transfer (APT) and the relayed nuclear Overhauser enhancement (rNOE). The results demonstrated that the CEST contrast maps derived from the CEST images of AMO-CEST were comparable to those derived from the original high-resolution CEST images. The proposed AMO-CEST can efficiently reconstruct high-quality CEST images from under-sampled k-space data and thus has the potential to accelerate CEST-MRI acquisition.

NeurIPS Conference 2024 Conference Paper

Eye-gaze Guided Multi-modal Alignment for Medical Representation Learning

• Chong Ma
• Hanqi Jiang
• Wenting Chen
• Yiwei Li
• Zihao Wu
• Xiaowei Yu
• Zhengliang Liu
• Lei Guo

In the medical multi-modal frameworks, the alignment of cross-modality features presents a significant challenge. However, existing works have learned features that are implicitly aligned from the data, without considering the explicit relationships in the medical context. This data-reliance may lead to low generalization of the learned alignment relationships. In this work, we propose the Eye-gaze Guided Multi-modal Alignment (EGMA) framework to harness eye-gaze data for better alignment of medical visual and textual features. We explore the natural auxiliary role of radiologists' eye-gaze data in aligning medical images and text, and introduce a novel approach by using eye-gaze data, collected synchronously by radiologists during diagnostic evaluations. We conduct downstream tasks of image classification and image-text retrieval on four medical datasets, where EGMA achieved state-of-the-art performance and stronger generalization across different datasets. Additionally, we explore the impact of varying amounts of eye-gaze data on model performance, highlighting the feasibility and utility of integrating this auxiliary data into multi-modal alignment framework.

JBHI Journal 2024 Journal Article

FEFA: Frequency Enhanced Multi-Modal MRI Reconstruction With Deep Feature Alignment

• Xuanmin Chen
• Liyan Ma
• Shihui Ying
• Dinggang Shen
• Tieyong Zeng

Integrating complementary information from multiple magnetic resonance imaging (MRI) modalities is often necessary to make accurate and reliable diagnostic decisions. However, the different acquisition speeds of these modalities mean that obtaining information can be time consuming and require significant effort. Reference-based MRI reconstruction aims to accelerate slower, under-sampled imaging modalities, such as T2-modality, by utilizing redundant information from faster, fully sampled modalities, such as T1-modality. Unfortunately, spatial misalignment between different modalities often negatively impacts the final results. To address this issue, we propose FEFA, which consists of cascading FEFA blocks. The FEFA block first aligns and fuses the two modalities at the feature level. The combined features are then filtered in the frequency domain to enhance the important features while simultaneously suppressing the less essential ones, thereby ensuring accurate reconstruction. Furthermore, we emphasize the advantages of combining the reconstruction results from multiple cascaded blocks, which also contributes to stabilizing the training process. Compared to existing registration-then-reconstruction and cross-attention-based approaches, our method is end-to-end trainable without requiring additional supervision, extensive parameters, or heavy computation. Experiments on the public fastMRI, IXI and in-house datasets demonstrate that our approach is effective across various under-sampling patterns and ratios.

JBHI Journal 2024 Journal Article

Image Recovery Matters: A Recovery-Extraction Framework for Robust Fetal Brain Extraction From MR Images

• Jian Chen
• Ranlin Lu
• Shilin Ye
• Mengting Guang
• Tewodros Megabiaw Tassew
• Bin Jing
• Guofu Zhang
• Geng Chen

The extraction of the fetal brain from magnetic resonance (MR) images is a challenging task. In particular, fetal MR images suffer from different kinds of artifacts introduced during the image acquisition. Among those artifacts, intensity inhomogeneity is a common one affecting brain extraction. In this work, we propose a deep learning-based recovery-extraction framework for fetal brain extraction, which is particularly effective in handling fetal MR images with intensity inhomogeneity. Our framework involves two stages. First, the artifact-corrupted images are recovered with the proposed generative adversarial learning-based image recovery network with a novel region-of-darkness discriminator that enforces the network focusing on artifacts of the images. Second, we propose a brain extraction network for more effective fetal brain segmentation by strengthening the association between lower- and higher-level features as well as suppressing task-irrelevant features. Thanks to the proposed recovery-extraction strategy, our framework is able to accurately segment fetal brains from artifact-corrupted MR images. The experiments show that our framework achieves promising performance in both quantitative and qualitative evaluations, and outperforms state-of-the-art methods in both image recovery and fetal brain extraction.

AAAI Conference 2024 Conference Paper

Mining Gaze for Contrastive Learning toward Computer-Assisted Diagnosis

• Zihao Zhao
• Sheng Wang
• Qian Wang
• Dinggang Shen

Obtaining large-scale radiology reports can be difficult for medical images due to ethical concerns, limiting the effectiveness of contrastive pre-training in the medical image domain and underscoring the need for alternative methods. In this paper, we propose eye-tracking as an alternative to text reports, as it allows for the passive collection of gaze signals without ethical issues. By tracking the gaze of radiologists as they read and diagnose medical images, we can understand their visual attention and clinical reasoning. When a radiologist has similar gazes for two medical images, it may indicate semantic similarity for diagnosis, and these images should be treated as positive pairs when pre-training a computer-assisted diagnosis (CAD) network through contrastive learning. Accordingly, we introduce the Medical contrastive Gaze Image Pre-training (McGIP) as a plug-and-play module for contrastive learning frameworks. McGIP uses radiologist gaze to guide contrastive pre-training. We evaluate our method using two representative types of medical images and two common types of gaze data. The experimental results demonstrate the practicality of McGIP, indicating its high potential for various clinical scenarios and applications.

AAAI Conference 2024 Conference Paper

NaMa: Neighbor-Aware Multi-Modal Adaptive Learning for Prostate Tumor Segmentation on Anisotropic MR Images

• Runqi Meng
• Xiao Zhang
• Shijie Huang
• Yuning Gu
• Guiqin Liu
• Guangyu Wu
• Nizhuan Wang
• Kaicong Sun

Accurate segmentation of prostate tumors from multi-modal magnetic resonance (MR) images is crucial for diagnosis and treatment of prostate cancer. However, the robustness of existing segmentation methods is limited, mainly because these methods 1) fail to adaptively assess subject-specific information of each MR modality for accurate tumor delineation, and 2) lack effective utilization of inter-slice information across thick slices in MR images to segment tumor as a whole 3D volume. In this work, we propose a two-stage neighbor-aware multi-modal adaptive learning network (NaMa) for accurate prostate tumor segmentation from multi-modal anisotropic MR images. In particular, in the first stage, we apply subject-specific multi-modal fusion in each slice by developing a novel modality-informativeness adaptive learning (MIAL) module for selecting and adaptively fusing informative representation of each modality based on inter-modality correlations. In the second stage, we exploit inter-slice feature correlations to derive volumetric tumor segmentation. Specifically, we first use a Unet variant with sequence layers to coarsely capture slice relationship at a global scale, and further generate an activation map for each slice. Then, we introduce an activation mapping guidance (AMG) module to refine slice-wise representation (via information from adjacent slices) for consistent tumor segmentation across neighboring slices. Besides, during the network training, we further apply a random mask strategy to each MR modality to improve feature representation efficiency. Experiments on both in-house and public (PICAI) multi-modal prostate tumor datasets show that our proposed NaMa performs better than state-of-the-art methods.

JBHI Journal 2024 Journal Article

Structure-Aware Registration Network for Liver DCE-CT Images

• Peng Xue
• Jingyang Zhang
• Lei Ma
• Mianxin Liu
• Yuning Gu
• Jiawei Huang
• Feihong Liu
• Yongsheng Pan

Image registration of liver dynamic contrast-enhanced computed tomography (DCE-CT) is crucial for diagnosis and image-guided surgical planning of liver cancer. However, intensity variations due to the flow of contrast agents combined with complex spatial motion induced by respiration brings great challenge to existing intensity-based registration methods. To address these problems, we propose a novel structure-aware registration method by incorporating structural information of related organs with segmentation-guided deep registration network. Existing segmentation-guided registration methods only focus on volumetric registration inside the paired organ segmentations, ignoring the inherent attributes of their anatomical structures. In addition, such paired organ segmentations are not always available in DCE-CT images due to the flow of contrast agents. Different from existing segmentation-guided registration methods, our proposed method extracts structural information in hierarchical geometric perspectives of line and surface. Then, according to the extracted structural information, structure-aware constraints are constructed and imposed on the forward and backward deformation field simultaneously. In this way, all available organ segmentations, including unpaired ones, can be fully utilized to avoid the side effect of contrast agent and preserve the topology of organs during registration. Extensive experiments on an in-house liver DCE-CT dataset and a public LiTS dataset show that our proposed method can achieve higher registration accuracy and preserve anatomical structure more effectively than state-of-the-art methods.

JBHI Journal 2023 Journal Article

Brain Status Transferring Generative Adversarial Network for Decoding Individualized Atrophy in Alzheimer's Disease

• Xingyu Gao
• Hongrui Liu
• Feng Shi
• Dinggang Shen
• Manhua Liu

Deep learning has been widely investigated in brain image computational analysis for diagnosing brain diseases such as Alzheimer's disease (AD). Most of the existing methods built end-to-end models to learn discriminative features by group-wise analysis. However, these methods cannot detect pathological changes in each subject, which is essential for the individualized interpretation of disease variances and precision medicine. In this article, we propose a brain status transferring generative adversarial network (BrainStatTrans-GAN) to generate corresponding healthy images of patients, which are further used to decode individualized brain atrophy. The BrainStatTrans-GAN consists of generator, discriminator, and status discriminator. First, a normative GAN is built to generate healthy brain images from normal controls. However, it cannot generate healthy images from diseased ones due to the lack of paired healthy and diseased images. To address this problem, a status discriminator with adversarial learning is designed in the training process to produce healthy brain images for patients. Then, the residual between the generated and input images can be computed to quantify pathological brain changes. Finally, a residual-based multi-level fusion network (RMFN) is built for more accurate disease diagnosis. Compared to the existing methods, our method can model individualized brain atrophy for facilitating disease diagnosis and interpretation. Experimental results on T1-weighted magnetic resonance imaging (MRI) data of 1, 739 subjects from three datasets demonstrate the effectiveness of our method.

JBHI Journal 2023 Journal Article

Individualized Assessment of Brain Aβ Deposition With fMRI Using Deep Learning

• Chaolin Li
• Mianxin Liu
• Jing Xia
• Lang Mei
• Qing Yang
• Feng Shi
• Han Zhang
• Dinggang Shen

PET-based Alzheimer's disease (AD) assessment has many limitations in large-scale screening. Non-invasive techniques such as resting-state functional magnetic resonance imaging (rs-fMRI) have been proven valuable in early AD diagnosis. This study investigated feasibility of using rs-fMRI, especially functional connectivity (FC), for individualized assessment of brain amyloid-β deposition derived from PET. We designed a graph convolutional networks (GCNs) and random forest (RF) based integrated framework for using rs-fMRI-derived multi-level FC networks to predict amyloid-β PET patterns with the OASIS-3 (N = 258) and ADNI-2 (N = 291) datasets. Our method achieved satisfactory accuracy not only in Aβ-PET grade classification (for negative, intermediate, and positive grades, with accuracy in the three-class classification as 62. 8% and 64. 3% on two datasets, respectively), but also in prediction of whole-brain region-level Aβ-PET standard uptake value ratios (SUVRs) (with the mean square errors as 0. 039 and 0. 074 for two datasets, respectively). Model interpretability examination also revealed the contributive role of the limbic network. This study demonstrated high feasibility and reproducibility of using low-cost, more accessible magnetic resonance imaging (MRI) to approximate PET-based diagnosis.

JBHI Journal 2023 Journal Article

Multi-Graph Attention Networks With Bilinear Convolution for Diagnosis of Schizophrenia

• Renping Yu
• Cong Pan
• Xuan Fei
• Mingming Chen
• Dinggang Shen

The explorations of brain functional connectivity (FC) network using resting-state functional magnetic resonance imaging (rs-fMRI) can provide crucial insights into discriminative analysis of neuropsychiatric disorders such as schizophrenia (SZ). Graph attention network (GAT), which could capture the local stationary on the network topology and aggregate the features of neighboring nodes, has advantages in learning the feature representation of brain regions. However, GAT only can obtain the node-level features that reflect local information, ignoring the spatial information within the connectivity-based features that proved to be important for SZ diagnosis. In addition, existing graph learning techniques usually rely on a single graph topology to represent neighborhood information, and only consider a single correlation measure for connectivity features. Comprehensive analysis of multiple graph topologies and multiple measures of FC can leverage their complementary information that may contribute to identifying patients. In this paper, we propose a multi-graph attention network (MGAT) with bilinear convolution (BC) neural network framework for SZ diagnosis and functional connectivity analysis. Besides multiple correlation measures to construct connectivity networks from different perspectives, we further propose two different graph construction methods to capture both the low- and high-level graph topologies, respectively. Especially, the MGAT module is developed to learn multiple node interaction features on each graph topology, and the BC module is utilized to learn the spatial connectivity features of the brain network for disease prediction. Importantly, the rationality and advantages of our proposed method can be validated by the experiments on SZ identification. Therefore, we speculate that this framework may also be potentially used as a diagnostic tool for other neuropsychiatric disorders.

JBHI Journal 2023 Journal Article

Outcome Prediction of Unconscious Patients Based on Weighted Sparse Brain Network Construction

• Renping Yu
• Han Zhang
• Xuehai Wu
• Xuan Fei
• Qing Yang
• Zhiwei Ma
• Zengxin Qi
• Di Zang

It is quite challenging to establish a prompt and reliable prognosis assessment for acquired brain injury (ABI) patients with persistent severe disorders of consciousness (DOC) like unconscious comatose and unresponsive wakefulness syndrome (a. k. a. , vegetative state). Recent advances in brain functional imaging and functional net-work analysis have demonstrated its potential in determining the consciousness level and prognostic outcome for ABI patients with DOC. However, the diagnostic and prognostic usefulness of the whole-brain functional connectome based on advanced machine learning techniques has not been fully evaluated. The first aim of this study is to predict the outcome of individual unconscious ABI patients during a three-month follow-up. The second aim is to conduct precise individualized differentiation among different consciousness levels for exploring the neurobiological mechanisms underlying DOC. Based on resting-state fMRI, we construct large-scale functional networks by using a weighted sparse model, which ensures sparsity and interpretability by preserving strong functional connections. The functional connection strengths are exploited as features for outcome prediction and consciousness level differentiation. We achieve significantly improved consciousness level classification (accuracy: 84. 78%) and recovery outcome prediction (accuracy: 89. 74%) compared to other network construction methods. More importantly, we reveal the contributive connections across the entire brain in both tasks. These connections could serve as the potential biomarkers for better understanding of consciousness and further provide new insight into the development of diagnostic, prognostic, and effective therapeutic guidelines for ABI patients with DOC.

JBHI Journal 2023 Journal Article

TW-Net: Transformer Weighted Network for Neonatal Brain MRI Segmentation

• Shengjie Zhang
• Bohan Ren
• Ziqi Yu
• Haibo Yang
• Xiaoyang Han
• Xiang Chen
• Yuan Zhou
• Dinggang Shen

Accurate neonatal brain MRI segmentation is valuable for investigating brain growth patterns and tracking the progression of neurodevelopmental disorders. However, it is a challenging task to use intensity-based methods to segment neonatal brain structures because of small contrast differences between brain regions caused by the inherent myelination process. Although convolutional neural networks offer the potential to segment brain structures in an intensity-independent manner, they suffer from lack of in-plane long-range dependency which is essential for the segmentation. To solve this problem, we propose a novel Transformer-Weighted network (TW-Net) to incorporate in-plane long-range dependency information. TW-Net employs a conventional encoder-decoder architecture with a Transformer module in the middle. The Transformer module uses a rotate-and-flip layer to better calculate the similarity between two patches in a slice to leverage similar patterns of geometrical and texture features within brain structures. In addition, a deep supervision module and squeeze-and-excitation blocks are introduced to incorporate boundary information of brain structures. Compared with state-of-the-art deep learning algorithms, TW-Net outperforms these methods for multiple-label tasks in 2D and 2. 5D configurations on two independent public datasets, demonstrating that TW-Net is a promising method for neonatal brain MRI segmentation.

JBHI Journal 2023 Journal Article

Two-Stage Self-Supervised Cycle-Consistency Transformer Network for Reducing Slice Gap in MR Images

• Zhiyang Lu
• Jian Wang
• Zheng Li
• Shihui Ying
• Jun Wang
• Jun Shi
• Dinggang Shen

Magnetic resonance (MR) images are usually acquired with large slice gap in clinical practice, i. e. , low resolution (LR) along the through-plane direction. It is feasible to reduce the slice gap and reconstruct high-resolution (HR) images with the deep learning (DL) methods. To this end, the paired LR and HR images are generally required to train a DL model in a popular fully supervised manner. However, since the HR images are hardly acquired in clinical routine, it is difficult to get sufficient paired samples to train a robust model. Moreover, the widely used convolutional Neural Network (CNN) still cannot capture long-range image dependencies to combine useful information of similar contents, which are often spatially far away from each other across neighboring slices. To this end, a Two-stage Self-supervised Cycle-consistency Transformer Network (TSCTNet) is proposed to reduce the slice gap for MR images in this work. A novel self-supervised learning (SSL) strategy is designed with two stages respectively for robust network pre-training and specialized network refinement based on a cycle-consistency constraint. A hybrid Transformer and CNN structure is utilized to build an interpolation model, which explores both local and global slice representations. The experimental results on two public MR image datasets indicate that TSCTNet achieves superior performance over other compared SSL-based algorithms.

YNICL Journal 2022 Journal Article

Alterations of dynamic redundancy of functional brain subnetworks in Alzheimer’s disease and major depression disorders

• Maryam Ghanbari
• Mayssa Soussia
• Weixiong Jiang
• Dongming Wei
• Pew-Thian Yap
• Dinggang Shen
• Han Zhang

The human brain is not only efficiently but also "redundantly" connected. The redundancy design could help the brain maintain resilience to disease attacks. This paper explores subnetwork-level redundancy dynamics and the potential of such metrics in disease studies. As such, we looked into specific functional subnetworks, including those associated with high-level functions. We investigated how the subnetwork redundancy dynamics change along with Alzheimer's disease (AD) progression and with major depressive disorder (MDD), two major disorders that could share similar subnetwork alterations. We found an increased dynamic redundancy of the subcortical-cerebellum subnetwork and its connections to other high-order subnetworks in the mild cognitive impairment (MCI) and AD compared to the normal control (NC). With gained spatial specificity, we found such a redundancy index was sensitive to disease symptoms and could act as a protective mechanism to prevent the collapse of the brain network and functions. The dynamic redundancy of the medial frontal subnetwork and its connections to the frontoparietal subnetwork was also found decreased in MDD compared to NC. The spatial specificity of the redundancy dynamics changes may provide essential knowledge for a better understanding of shared neural substrates in AD and MDD.

JBHI Journal 2022 Journal Article

Cross-Model Attention-Guided Tumor Segmentation for 3D Automated Breast Ultrasound (ABUS) Images

• Yue Zhou
• Houjin Chen
• Yanfeng Li
• Xuyang Cao
• Shu Wang
• Dinggang Shen

Tumor segmentation in 3D automated breast ultrasound (ABUS) plays an important role in breast disease diagnosis and surgical planning. However, automatic segmentation of tumors in 3D ABUS images is still challenging, due to the large tumor shape and size variations, and uncertain tumor locations among patients. In this paper, we develop a novel cross-model attention-guided tumor segmentation network with a hybrid loss for 3D ABUS images. Specifically, we incorporate the tumor location into a segmentation network by combining an improved 3D Mask R-CNN head into V-Net as an end-to-end architecture. Furthermore, we introduce a cross-model attention mechanism that is able to aggregate the segmentation probability map from the improved 3D Mask R-CNN to each feature extraction level in the V-Net. Then, we design a hybrid loss to balance the contribution of each part in the proposed cross-model segmentation network. We conduct extensive experiments on 170 3D ABUS from 107 patients. Experimental results show that our method outperforms other state-of-the-art methods, by achieving the Dice similarity coefficient (DSC) of 64. 57%, Jaccard coefficient (JC) of 53. 39%, recall (REC) of 64. 43%, precision (PRE) of 74. 51%, 95th Hausdorff distance (95HD) of 11. 91 mm, and average surface distance (ASD) of 4. 63 mm. Our code will be available online ( https://github.com/zhouyuegithub/CMVNet ).

YNIMG Journal 2022 Journal Article

Deep attentive spatio-temporal feature learning for automatic resting-state fMRI denoising

• Keun-Soo Heo
• Dong-Hee Shin
• Sheng-Che Hung
• Weili Lin
• Han Zhang
• Dinggang Shen
• Tae-Eui Kam

Resting-state functional magnetic resonance imaging (rs-fMRI) is a non-invasive functional neuroimaging modality that has been widely used to investigate functional connectomes in the brain. Since noise and artifacts generated by non-neuronal physiological activities are predominant in raw rs-fMRI data, effective noise removal is one of the most important preprocessing steps prior to any subsequent analysis. For rs-fMRI denoising, a common trend is to decompose rs-fMRI data into multiple components and then regress out noise-related components. Therefore, various machine learning techniques have been used in such analyses with predefined procedures and manually engineered features. However, the lack of a universal definition of a noise-related source or artifact complicates manual feature engineering. Manual feature selection can result in the failure to capture unknown types of noise. Furthermore, the possibility that the hand-crafted features will only work for the broader population (e.g., healthy adults) but not for "outliers" (e.g., infants or subjects that belong to a disease cohort) is quite high. In practice, we have limited knowledge of which features should be extracted; thus, multi-classifier assembly must be implemented to improve performance, although this process is quite time-consuming. However, in real rs-fMRI applications, fast and accurate automatic identification of noise-related components on different datasets is critical. To solve this problem, we propose a novel, automatic, and end-to-end deep learning framework dedicated to noise-related component identification via a faster and more effective multi-layer feature extraction strategy that learns deeply embedded spatio-temporal features of the components. In this study, we achieved remarkable performance on various rs-fMRI datasets, including multiple adult rs-fMRI datasets from different rs-fMRI studies and an infant rs-fMRI dataset, which is quite heterogeneous and differs from that of adults. Our proposed framework also dramatically increases the noise detection speed owing to its inherent ability for deep learning (< 1s for single-component classification). It can be easily integrated into any preprocessing pipeline, even those that do not use standard procedures but depend on alternative toolboxes.

JBHI Journal 2022 Journal Article

Divergent and Convergent Imaging Markers Between Bipolar and Unipolar Depression Based on Machine Learning

• Huifeng Zhang
• Zhen Zhou
• Lei Ding
• Chuangxin Wu
• Meihui Qiu
• Yueqi Huang
• Feng Jin
• Ting Shen

Distinguishing bipolar depression (BD) from unipolar depression (UD) based on symptoms only is challenging. Brain functional connectivity (FC), especially dynamic FC, has emerged as a promising approach to identify possible imaging markers for differentiating BD from UD. However, most of such studies utilized conventional FC and group-level statistical comparisons, which may not be sensitive enough to quantify subtle changes in the FC dynamics between BD and UD. In this paper, we present a more effective individualized differentiation model based on machine learning and the whole-brain “high-order functional connectivity (HOFC)” network. The HOFC, capturing temporal synchronization among the dynamic FC time series, a more complex “chronnectome” metric compared to the conventional FC, was used to classify 52 BD, 73 UD, and 76 healthycontrols (HC). We achieved a satisfactory accuracy (70. 40%) in BD vs. UD differentiation. The resultant contributing features revealed the involvement of the coordinated flexible interactions among sensory (e. g. , olfaction, vision, and audition), motor, and cognitive systems. Despite sharing common chronnectome of cognitive and affective impairments, BD and UD also demonstrated unique dynamic FC synchronization patterns. UD is more associated with abnormal visual-somatomotor inter-network connections, while BD is more related to impaired ventral attention-frontoparietal inter-network connections. Moreover, we found that the illness duration modulated the BD vs. UD separation, with the differentiation performance hampered by the secondary disease effects. Our findings suggest that BD and UD may have divergent and convergent neural substrates, which further expand our knowledge of the two different mental disorders.

NeurIPS Conference 2022 Conference Paper

Forecasting Human Trajectory from Scene History

• Mancheng Meng
• Ziyan Wu
• Terrence Chen
• Xiran Cai
• Xiang Zhou
• Fan Yang
• Dinggang Shen

Predicting the future trajectory of a person remains a challenging problem, due to randomness and subjectivity. However, the moving patterns of human in constrained scenario typically conform to a limited number of regularities to a certain extent, because of the scenario restrictions (\eg, floor plan, roads and obstacles) and person-person or person-object interactivity. Thus, an individual person in this scenario should follow one of the regularities as well. In other words, a person's subsequent trajectory has likely been traveled by others. Based on this hypothesis, we propose to forecast a person's future trajectory by learning from the implicit scene regularities. We call the regularities, inherently derived from the past dynamics of the people and the environment in the scene, \emph{scene history}. We categorize scene history information into two types: historical group trajectories and individual-surroundings interaction. To exploit these information for trajectory prediction, we propose a novel framework Scene History Excavating Network (SHENet), where the scene history is leveraged in a simple yet effective approach. In particular, we design two components, the group trajectory bank module to extract representative group trajectories as the candidate for future path, and the cross-modal interaction module to model the interaction between individual past trajectory and its surroundings for trajectory refinement, respectively. In addition, to mitigate the uncertainty in the evaluation, caused by the aforementioned randomness and subjectivity, we propose to include smoothness into evaluation metrics. We conduct extensive evaluations to validate the efficacy of proposed framework on ETH, UCY, as well as a new, challenging benchmark dataset PAV, demonstrating superior performance compared to state-of-the-art methods.

JBHI Journal 2022 Journal Article

GAN-Guided Deformable Attention Network for Identifying Thyroid Nodules in Ultrasound Images

• Jintao Lu
• Xi Ouyang
• Xueda Shen
• Tianjiao Liu
• Zhiming Cui
• Qian Wang
• Dinggang Shen

Early detection and identification of malignant thyroid nodules, a vital precursory to the treatment, is a difficult task even for experienced clinicians. Many Computer-Aided Diagnose (CAD) systems have been developed to assist clinicians in performing this task on ultrasonic images. Learning-based CAD systems for thyroid nodules generally accommodate both nodule detection/ segmentation and fine-grained classification for its malignancy, and prior researches often treat aforementioned tasks in separate stages, leading to additional computational costs. In this paper, we utilize an online class activation mapping (CAM) mechanism to guide the network to learn discriminative features for identifying thyroid nodules in ultrasound images, called CAM attention network. It takes nodule masks as localization cues for direct spatial attention of the classification module, thereby avoiding isolated training for classification. Meanwhile, we propose a deformable convolution module to add offsets to the regular grid sampling locations in the standard convolution, guiding the network to capture more discriminative features of nodule areas. Furthermore, we use a generative adversarial network (GAN)to ensure reliable deformations of nodules from the deformable convolution module. Our proposed CAM attention network has already achieved the 2nd place in the classification task of TN-SCUI 2020, a MICCAI 2020 Challenge with the largest set of thyroid nodule ultrasound images according to our knowledge. The further inclusion of our proposed GAN-guided deformable module allows for capturing more fine-grained features between benign and malignant nodules, and further improves the classification accuracy to a new state-of-the-art level.

JBHI Journal 2022 Journal Article

Multiple B-Value Model-Based Residual Network (MORN) for Accelerated High-Resolution Diffusion-Weighted Imaging

• Fanwen Wang
• Hui Zhang
• Fei Dai
• Weibo Chen
• Shuai Xu
• Zidong Yang
• Dinggang Shen
• Chengyan Wang

Single-Shot Echo Planar Imaging (SSEPI) based Diffusion Weighted Imaging (DWI) has shortcomings such as low resolution and severe distortions. In contrast, Multi-Shot EPI (MSEPI) provides optimal spatial resolution but increases scan time. This study proposed a Multiple b-value mOdel-based Residual Network (MORN) model to reconstruct multiple b-value high-resolution DWI from undersampled k-space data simultaneously. We incorporated Parallel Imaging (PI) into a residual U-net to reconstruct multiple b-value multi-coil data with the supervision of MUltiplexed Sensitivity-Encoding (MUSE) reconstructed Multi-Shot DWI (MSDWI). Moreover, asymmetric concatenations among different b-values and the combined loss to back propagate helped the feature transfer. After training and validation of the MORN in a dataset of 32 healthy cases, additional assessments were performed on 6 patients with different tumor types. The experimental results demonstrated that the MORN model outperformed conventional PI reconstruction (i. e. SENSE) and two state-of-the-art deep learning methods (SENSE-GAN and VSNet) in terms of PSNR (Peak Signal-to-Noise Ratio), SSIM (Structual SIMilarity) and apparent diffusion coefficient maps. In addition, using the pre-trained model under DWI, the MORN achieved consistent fractional anisotrophy and mean diffusivity reconstructed from multiple diffusion directions. Hence, the proposed method shows potential in clinical application according to the observations on tumor patients as well as images of multiple diffusion directions.

JBHI Journal 2022 Journal Article

Task-Induced Pyramid and Attention GAN for Multimodal Brain Image Imputation and Classification in Alzheimer's Disease

• Xingyu Gao
• Feng Shi
• Dinggang Shen
• Manhua Liu

With the advance of medical imaging technologies, multimodal images such as magnetic resonance images (MRI) and positron emission tomography (PET) can capture subtle structural and functional changes of brain, facilitating the diagnosis of brain diseases such as Alzheimer's disease (AD). In practice, multimodal images may be incomplete since PET is often missing due to high financial costs or availability. Most of the existing methods simply excluded subjects with missing data, which unfortunately reduced the sample size. In addition, how to extract and combine multimodal features is still challenging. To address these problems, we propose a deep learning framework to integrate a task-induced pyramid and attention generative adversarial network (TPA-GAN) with a pathwise transfer dense convolution network (PT-DCN) for imputation and classification of multimodal brain images. First, we propose a TPA-GAN to integrate pyramid convolution and attention module as well as disease classification task into GAN for generating the missing PET data with their MRI. Then, with the imputed multimodal images, we build a dense convolution network with pathwise transfer blocks to gradually learn and combine multimodal features for final disease classification. Experiments are performed on ADNI-1/2 datasets to evaluate our method, achieving superior performance in image imputation and brain disease diagnosis compared to state-of-the-art methods.

YNIMG Journal 2021 Journal Article

DIKA-Nets: Domain-invariant knowledge-guided attention networks for brain skull stripping of early developing macaques

• Tao Zhong
• Fenqiang Zhao
• Yuchen Pei
• Zhenyuan Ning
• Lufan Liao
• Zhengwang Wu
• Yuyu Niu
• Li Wang

As non-human primates, macaques have a close phylogenetic relationship to human beings and have been proven to be a valuable and widely used animal model in human neuroscience research. Accurate skull stripping (aka. brain extraction) of brain magnetic resonance imaging (MRI) is a crucial prerequisite in neuroimaging analysis of macaques. Most of the current skull stripping methods can achieve satisfactory results for human brains, but when applied to macaque brains, especially during early brain development, the results are often unsatisfactory. In fact, the early dynamic, regionally-heterogeneous development of macaque brains, accompanied by poor and age-related contrast between different anatomical structures, poses significant challenges for accurate skull stripping. To overcome these challenges, we propose a fully-automated framework to effectively fuse the age-specific intensity information and domain-invariant prior knowledge as important guiding information for robust skull stripping of developing macaques from 0 to 36 months of age. Specifically, we generate Signed Distance Map (SDM) and Center of Gravity Distance Map (CGDM) based on the intermediate segmentation results as guidance. Instead of using local convolution, we fuse all information using the Dual Self-Attention Module (DSAM), which can capture global spatial and channel-dependent information of feature maps. To extensively evaluate the performance, we adopt two relatively-large challenging MRI datasets from rhesus macaques and cynomolgus macaques, respectively, with a total of 361 scans from two different scanners with different imaging protocols. We perform cross-validation by using one dataset for training and the other one for testing. Our method outperforms five popular brain extraction tools and three deep-learning-based methods on cross-source MRI datasets without any transfer learning.

AIIM Journal 2021 Journal Article

Interactive medical image segmentation via a point-based interaction

• Jian Zhang
• Yinghuan Shi
• Jinquan Sun
• Lei Wang
• Luping Zhou
• Yang Gao
• Dinggang Shen

Due to low tissue contrast, irregular shape, and large location variance, segmenting the objects from different medical imaging modalities (e. g. , CT, MR) is considered as an important yet challenging task. In this paper, a novel method is presented for interactive medical image segmentation with the following merits. (1) Its design is fundamentally different from previous pure patch-based and image-based segmentation methods. It is observed that during delineation, the physician repeatedly check the intensity from area inside-object to outside-object to determine the boundary, which indicates that comparison in an inside-out manner is extremely important. Thus, the method innovatively models the segmentation task as learning the representation of bi-directional sequential patches, starting from (or ending in) the given central point of the object. This can be realized by the proposed ConvRNN network embedded with a gated memory propagation unit. (2) Unlike previous interactive methods (requiring bounding box or seed points), the proposed method only asks the physician to merely click on the rough central point of the object before segmentation, which could simultaneously enhance the performance and reduce the segmentation time. (3) The method is utilized in a multi-level framework for better performance. It has been systematically evaluated in three different segmentation tasks, including CT kidney tumor, MR prostate, and PROMISE12 challenge, showing promising results compared with state-of-the-art methods.

JBHI Journal 2021 Journal Article

Learning-Based Computer-Aided Prescription Model for Parkinson's Disease: A Data-Driven Perspective

• Yinghuan Shi
• Wanqi Yang
• Kim-Han Thung
• Hao Wang
• Yang Gao
• Yang Pan
• Li Zhang
• Dinggang Shen

In this article, we study a novel problem: “automatic prescription recommendation for PD patients. ” To realize this goal, we first build a dataset by collecting 1) symptoms of PD patients, and 2) their prescription drug provided by neurologists. Then, we build a novel computer-aided prescription model by learning the relation between observed symptoms and prescription drug. Finally, for the new coming patients, we could recommend (predict) suitable prescription drug on their observed symptoms by our prescription model. From the methodology part, our proposed model, namely Prescription viA Learning lAtent Symptoms (PALAS), could recommend prescription using the multi-modality representation of the data. In PALAS, a latent symptom space is learned to better model the relationship between symptoms and prescription drug, as there is a large semantic gap between them. Moreover, we present an efficient alternating optimization method for PALAS. We evaluated our method using the data collected from 136 PD patients at Nanjing Brain Hospital, which can be regarded as a large dataset in PD research community. The experimental results demonstrate the effectiveness and clinical potential of our method in this recommendation task, if compared with other competing methods.

YNICL Journal 2021 Journal Article

Multiscale neural modeling of resting-state fMRI reveals executive-limbic malfunction as a core mechanism in major depressive disorder

• Guoshi Li
• Yujie Liu
• Yanting Zheng
• Ye Wu
• Danian Li
• Xinyu Liang
• Yaoping Chen
• Ying Cui

Major depressive disorder (MDD) represents a grand challenge to human health and society, but the underlying pathophysiological mechanisms remain elusive. Previous neuroimaging studies have suggested that MDD is associated with abnormal interactions and dynamics in two major neural systems including the default mode - salience (DMN-SAL) network and the executive - limbic (EXE-LIM) network, but it is not clear which network plays a central role and which network plays a subordinate role in MDD pathophysiology. To address this question, we refined a newly developed Multiscale Neural Model Inversion (MNMI) framework and applied it to test whether MDD is more affected by impaired circuit interactions in the DMN-SAL network or the EXE-LIM network. The model estimates the directed connection strengths between different neural populations both within and between brain regions based on resting-state fMRI data collected from normal healthy subjects and patients with MDD. Results show that MDD is primarily characterized by abnormal circuit interactions in the EXE-LIM network rather than the DMN-SAL network. Specifically, we observe reduced frontoparietal effective connectivity that potentially contributes to hypoactivity in the dorsolateral prefrontal cortex (dlPFC), and decreased intrinsic inhibition combined with increased excitation from the superior parietal cortex (SPC) that potentially lead to amygdala hyperactivity, together resulting in activation imbalance in the PFC-amygdala circuit that pervades in MDD. Moreover, the model reveals reduced PFC-to-hippocampus excitation but decreased SPC-to-thalamus inhibition in MDD population that potentially lead to hypoactivity in the hippocampus and hyperactivity in the thalamus, consistent with previous experimental data. Overall, our findings provide strong support for the long-standing limbic-cortical dysregulation model in major depression but also offer novel insights into the multiscale pathophysiology of this debilitating disease.

JBHI Journal 2020 Journal Article

Adaptive Feature Selection Guided Deep Forest for COVID-19 Classification With Chest CT

• Liang Sun
• Zhanhao Mo
• Fuhua Yan
• Liming Xia
• Fei Shan
• Zhongxiang Ding
• Bin Song
• Wanchun Gao

Chest computed tomography (CT) becomes an effective tool to assist the diagnosis of coronavirus disease-19 (COVID-19). Due to the outbreak of COVID-19 worldwide, using the computed-aided diagnosis technique for COVID-19 classification based on CT images could largely alleviate the burden of clinicians. In this paper, we propose an A daptive F eature S election guided D eep F orest (AFS-DF) for COVID-19 classification based on chest CT images. Specifically, we first extract location-specific features from CT images. Then, in order to capture the high-level representation of these features with the relatively small-scale data, we leverage a deep forest model to learn high-level representation of the features. Moreover, we propose a feature selection method based on the trained deep forest model to reduce the redundancy of features, where the feature selection could be adaptively incorporated with the COVID-19 classification model. We evaluated our proposed AFS-DF on COVID-19 dataset with 1495 patients of COVID-19 and 1027 patients of community acquired pneumonia (CAP). The accuracy (ACC), sensitivity (SEN), specificity (SPE), AUC, precision and F1-score achieved by our method are 91. 79%, 93. 05%, 89. 95%, 96. 35%, 93. 10% and 93. 07%, respectively. Experimental results on the COVID-19 dataset suggest that the proposed AFS-DF achieves superior performance in COVID-19 vs. CAP classification, compared with 4 widely used machine learning methods.

JBHI Journal 2020 Journal Article

An Effective MR-Guided CT Network Training for Segmenting Prostate in CT Images

• Wanqi Yang
• Yinghuan Shi
• Sang Hyun Park
• Ming Yang
• Yang Gao
• Dinggang Shen

Segmentation of prostate in medical imaging data (e. g. , CT, MRI, TRUS) is often considered as a critical yet challenging task for radiotherapy treatment. It is relatively easier to segment prostate from MR images than from CT images, due to better soft tissue contrast of the MR images. For segmenting prostate from CT images, most previous methods mainly used CT alone, and thus their performances are often limited by low tissue contrast in the CT images. In this article, we explore the possibility of using indirect guidance from MR images for improving prostate segmentation in the CT images. In particular, we propose a novel deep transfer learning approach, i. e. , MR-guided CT network training (namely MICS-NET), which can employ MR images to help better learning of features in CT images for prostate segmentation. In MICS-NET, the guidance from MRI consists of two steps: (1) learning informative and transferable features from MRI and then transferring them to CT images in a cascade manner, and (2) adaptively transferring the prostate likelihood of MRI model (i. e. , well-trained convnet by purely using MR images) with a view consistency constraint. To illustrate the effectiveness of our approach, we evaluate MICS-NET on a real CT prostate image set, with the manual delineations available as the ground truth for evaluation. Our methods generate promising segmentation results which achieve (1) six percentages higher Dice Ratio than the CT model purely using CT images and (2) comparable performance with the MRI model purely using MR images.

AAAI Conference 2020 Conference Paper

Auto-GAN: Self-Supervised Collaborative Learning for Medical Image Synthesis

• Bing Cao
• Han Zhang
• Nannan Wang
• Xinbo Gao
• Dinggang Shen

In various clinical scenarios, medical image is crucial in disease diagnosis and treatment. Different modalities of medical images provide complementary information and jointly helps doctors to make accurate clinical decision. However, due to clinical and practical restrictions, certain imaging modalities may be unavailable nor complete. To impute missing data with adequate clinical accuracy, here we propose a framework called self-supervised collaborative learning to synthesize missing modality for medical images. The proposed method comprehensively utilize all available information correlated to the target modality from multi-source-modality images to generate any missing modality in a single model. Different from the existing methods, we introduce an autoencoder network as a novel, self-supervised constraint, which provides target-modality-specific information to guide generator training. In addition, we design a modality mask vector as the target modality label. With experiments on multiple medical image databases, we demonstrate a great generalization ability as well as specialty of our method compared with other state-of-the-arts.

JBHI Journal 2020 Journal Article

Hierarchical Rough-to-Fine Model for Infant Age Prediction Based on Cortical Features

• Dan Hu
• Zhengwang Wu
• Weili Lin
• Gang Li
• Dinggang Shen

Prediction of the chronological age based on neuroimaging data is important for brain development analysis and brain disease diagnosis. Although many researches have been conducted for age prediction of older children and adults, little work has been dedicated to infants. To this end, this paper focuses on predicting infant age from birth to 2-year old using brain MR images, as well as identifying some related biomarkers. However, brain development during infancy is too rapid and heterogeneous to be accurately modeled by the conventional regression models. To address this issue, a two-stage prediction method is proposed. Specifically, our method first roughly predicts the age range of an infant and then finely predicts the accurate chronological age based on a learned, age-group-specific regression model. Combining this two-stage prediction method with another complementary one-stage prediction method, a hierarchical rough-to-fine (HRtoF) model is built. HRtoF effectively splits the rapid and heterogeneous changes during a long time period into several short time ranges and further mines the discrimination capability of cortical features, thus reaching high accuracy in infant age prediction. Taking 8 types of cortical morphometric features from structural MRI as predictors, the effectiveness of our proposed HRtoF model is validated using an infant dataset including 50 healthy subjects with 251 longitudinal MRI scans from 14 to 797 days. Comparing with five state-of-the-art regression methods, HRtoF model reduces the mean absolute error of the prediction from >48 days to 32. 1 days. The correlation coefficient of the predicted age and the chronological age reaches 0. 963. Moreover, based on HRtoF, the relative contributions of the eight types of cortical features for age prediction are also studied.

YNIMG Journal 2020 Journal Article

High-resolution 3D MR Fingerprinting using parallel imaging and deep learning

• Yong Chen
• Zhenghan Fang
• Sheng-Che Hung
• Wei-Tang Chang
• Dinggang Shen
• Weili Lin

MR Fingerprinting (MRF) is a relatively new imaging framework capable of providing accurate and simultaneous quantification of multiple tissue properties for improved tissue characterization and disease diagnosis. While 2D MRF has been widely available, extending the method to 3D MRF has been an actively pursued area of research as a 3D approach can provide a higher spatial resolution and better tissue characterization with an inherently higher signal-to-noise ratio. However, 3D MRF with a high spatial resolution requires lengthy acquisition times, especially for a large volume, making it impractical for most clinical applications. In this study, a high-resolution 3D MR Fingerprinting technique, combining parallel imaging and deep learning, was developed for rapid and simultaneous quantification of T1 and T2 relaxation times. Parallel imaging was first applied along the partition-encoding direction to reduce the amount of acquired data. An advanced convolutional neural network was then integrated with the MRF framework to extract features from the MRF signal evolution for improved tissue characterization and accelerated mapping. A modified 3D-MRF sequence was also developed in the study to acquire data to train the deep learning model that can be directly applied to prospectively accelerate 3D MRF scans. Our results of quantitative T1 and T2 maps demonstrate that improved tissue characterization can be achieved using the proposed method as compared to prior methods. With the integration of parallel imaging and deep learning techniques, whole-brain (26 × 26 × 18 cm3) quantitative T1 and T2 mapping with 1-mm isotropic resolution were achieved in ~7 min. In addition, a ~7-fold improvement in processing time to extract tissue properties was also accomplished with the deep learning approach as compared to the standard template matching method. All of these improvements make high-resolution whole-brain quantitative MR imaging feasible for clinical applications.

YNIMG Journal 2020 Journal Article

Morphology of perivascular spaces and enclosed blood vessels in young to middle-aged healthy adults at 7T: Dependences on age, brain region, and breathing gas

• Xiaopeng Zong
• Chunfeng Lian
• Jordan Jimenez
• Koji Yamashita
• Dinggang Shen
• Weili Lin

Perivascular spaces (PVSs) are fluid-filled spaces surrounding penetrating blood vessels in the brain and are an integral pathway of the glymphatic system. A PVS and the enclosed blood vessel are commonly visualized as a single vessel-like complex (denoted as PVSV) in high-resolution MRI images. Quantitative characterization of the PVSV morphology in MRI images in healthy subjects may serve as a reference for detecting disease related PVS and/or blood vessel alterations in patients with brain diseases. To this end, we evaluated the age dependences, spatial heterogeneities, and dynamic properties of PVSV morphological features in 45 healthy subjects (21–55 years old), using an ultra-high-resolution three-dimensional transverse relaxation time weighted MRI sequence (0. 41 ​× ​0. 41 ​× ​0. 4 ​mm3) at 7T. Quantitative PVSV parameters, including apparent diameter, count, volume fraction (VF), and relative contrast to noise ratio (rCNR) were calculated in the white matter and subcortical structures. Dynamic changes were induced by carbogen breathing which are known to induce vasodilation and increase the blood oxygenation level in the brain. PVSV count and VF significantly increased with age in basal ganglia (BG), so did rCNR in BG, midbrain, and white matter (WM). Apparent PVSV diameter also showed a positive association with age in the three brain regions, although it did not reach statistical significance. The PVSV VF and count showed large inter-subject variations, with coefficients of variation ranging from 0. 17 to 0. 74 after regressing out age and gender effects. Both apparent diameter and VF exhibited significant spatial heterogeneity, which cannot be explained solely by radio-frequency field inhomogeneities. Carbogen breathing significantly increased VF in BG and WM, and rCNR in thalamus, BG, and WM compared to air breathing. Our results are consistent with gradual dilation of PVSs with age in healthy adults. The PVSV morphology exhibited spatial heterogeneity and large inter-subject variations and changed during carbogen breathing compared to air breathing.

AAAI Conference 2020 Conference Paper

Non-Local U-Nets for Biomedical Image Segmentation

• Zhengyang Wang
• Na Zou
• Dinggang Shen
• Shuiwang Ji

Deep learning has shown its great promise in various biomedical image segmentation tasks. Existing models are typically based on U-Net and rely on an encoder-decoder architecture with stacked local operators to aggregate long-range information gradually. However, only using the local operators limits the efficiency and effectiveness. In this work, we propose the non-local U-Nets, which are equipped with flexible global aggregation blocks, for biomedical image segmentation. These blocks can be inserted into U-Net as size-preserving processes, as well as down-sampling and up-sampling layers. We perform thorough experiments on the 3D multimodality isointense infant brain MR image segmentation task to evaluate the non-local U-Nets. Results show that our proposed models achieve top performances with fewer parameters and faster computation.

JBHI Journal 2020 Journal Article

Toward a Better Estimation of Functional Brain Network for Mild Cognitive Impairment Identification: A Transfer Learning View

• Weikai Li
• Limei Zhang
• Lishan Qiao
• Dinggang Shen

Mild cognitive impairment (MCI) is an intermediate stage of brain cognitive decline, associated with increasing risk of developing Alzheimer's disease (AD). It is believed that early treatment of MCI could slow down the progression of AD, and functional brain network (FBN) could provide potential imaging biomarkers for MCI diagnosis and response to treatment. However, there are still some challenges to estimate a “good” FBN, particularly due to the poor quality and limited quantity of functional magnetic resonance imaging (fMRI) data from the target domain (i. e. , MCI study). Inspired by the idea of transfer learning, we attempt to transfer information in high-quality data from source domain (e. g. , human connectome project in this paper) into the target domain towards a better FBN estimation, and propose a novel method, namely NERTL (Network Estimation via Regularized Transfer Learning). Specifically, we first construct a high-quality network “template” based on the source data, and then use the template to guide or constrain the target of FBN estimation by a weighted l 1 -norm regularizer. Finally, we conduct experiments to identify subjects with MCI from normal controls (NCs) based on the estimated FBNs. Despite its simplicity, our proposed method is more effective than the baseline methods in modeling discriminative FBNs, as demonstrated by the superior MCI classification accuracy of 82. 4% and the area under curve (AUC) of 0. 910.

YNIMG Journal 2019 Journal Article

Computational neuroanatomy of baby brains: A review

• Gang Li
• Li Wang
• Pew-Thian Yap
• Fan Wang
• Zhengwang Wu
• Yu Meng
• Pei Dong
• Jaeil Kim

The first postnatal years are an exceptionally dynamic and critical period of structural, functional and connectivity development of the human brain. The increasing availability of non-invasive infant brain MR images provides unprecedented opportunities for accurate and reliable charting of dynamic early brain developmental trajectories in understanding normative and aberrant growth. However, infant brain MR images typically exhibit reduced tissue contrast (especially around 6 months of age), large within-tissue intensity variations, and regionally-heterogeneous, dynamic changes, in comparison with adult brain MR images. Consequently, the existing computational tools developed typically for adult brains are not suitable for infant brain MR image processing. To address these challenges, many infant-tailored computational methods have been proposed for computational neuroanatomy of infant brains. In this review paper, we provide a comprehensive review of the state-of-the-art computational methods for infant brain MRI processing and analysis, which have advanced our understanding of early postnatal brain development. We also summarize publically available infant-dedicated resources, including MRI datasets, computational tools, grand challenges, and brain atlases. Finally, we discuss the limitations in current research and suggest potential future research directions.

IJCAI Conference 2019 Conference Paper

Decoding EEG by Visual-guided Deep Neural Networks

• Zhicheng Jiao
• Haoxuan You
• Fan Yang
• Xin Li
• Han Zhang
• Dinggang Shen

Decoding visual stimuli from brain activities is an interdisciplinary study of neuroscience and computer vision. With the emerging of Human-AI Collaboration, Human-Computer Interaction, and the development of advanced machine learning models, brain decoding based on deep learning attracts more attention. Electroencephalogram (EEG) is a widely used neurophysiology tool. Inspired by the success of deep learning on image representation and neural decoding, we proposed a visual-guided EEG decoding method that contains a decoding stage and a generation stage. In the classification stage, we designed a visual-guided convolutional neural network (CNN) to obtain more discriminative representations from EEG, which are applied to achieve the classification results. In the generation stage, the visual-guided EEG features are input to our improved deep generative model with a visual consistence module to generate corresponding visual stimuli. With the help of our visual-guided strategies, the proposed method outperforms traditional machine learning methods and deep learning models in the EEG decoding task.

AAAI Conference 2019 Conference Paper

Difficulty-Aware Attention Network with Confidence Learning for Medical Image Segmentation

• Dong Nie
• Li Wang
• Lei Xiang
• Sihang Zhou
• Ehsan Adeli
• Dinggang Shen

Medical image segmentation is a key step for various applications, such as image-guided radiation therapy and diagnosis. Recently, deep neural networks provided promising solutions for automatic image segmentation; however, they often perform good on regular samples (i. e. , easy-to-segment samples), since the datasets are dominated by easy and regular samples. For medical images, due to huge inter-subject variations or disease-specific effects on subjects, there exist several difficult-to-segment cases that are often overlooked by the previous works. To address this challenge, we propose a difficulty-aware deep segmentation network with confidence learning for end-to-end segmentation. The proposed framework has two main contributions: 1) Besides the segmentation network, we also propose a fully convolutional adversarial network for confidence learning to provide voxel-wise and region-wise confidence information for the segmentation network. We relax the adversarial learning to confidence learning by decreasing the priority of adversarial learning, so that we can avoid the training imbalance between generator and discriminator. 2) We propose a difficulty-aware attention mechanism to properly handle hard samples or hard regions considering structural information, which may go beyond the shortcomings of focal loss. We further propose a fusion module to selectively fuse the concatenated feature maps in encoder-decoder architectures. Experimental results on clinical and challenge datasets show that our proposed network can achieve state-of-the-art segmentation accuracy. Further analysis also indicates that each individual component of our proposed network contributes to the overall performance improvement.

YNIMG Journal 2019 Journal Article

Exploring folding patterns of infant cerebral cortex based on multi-view curvature features: Methods and applications

• Dingna Duan
• Shunren Xia
• Islem Rekik
• Yu Meng
• Zhengwang Wu
• Li Wang
• Weili Lin
• John H. Gilmore

The highly convoluted cortical folding of the human brain is intriguingly complex and variable across individuals. Exploring the underlying representative patterns of cortical folding is of great importance for many neuroimaging studies. At term birth, all major cortical folds are established and are minimally affected by the complicated postnatal environments; hence, neonates are the ideal candidates for exploring early postnatal cortical folding patterns, which yet remain largely unexplored. In this paper, we propose a novel method for exploring the representative regional folding patterns of infant brains. Specifically, first, multi-view curvature features are constructed to comprehensively characterize the complex characteristics of cortical folding. Second, for each view of curvature features, a similarity matrix is computed to measure the similarity of cortical folding in a specific region between any pair of subjects. Next, a similarity network fusion method is adopted to nonlinearly and adaptively fuse all the similarity matrices into a single one for retaining both shared and complementary similarity information of the multiple characteristics of cortical folding. Finally, based on the fused similarity matrix and a hierarchical affinity propagation clustering approach, all subjects are automatically grouped into several clusters to obtain the representative folding patterns. To show the applications, we have applied the proposed method to a large-scale dataset with 595 normal neonates and discovered representative folding patterns in several cortical regions, i. e. , the superior temporal gyrus (STG), inferior frontal gyrus (IFG), precuneus, and cingulate cortex. Meanwhile, we have revealed sex difference in STG, IFG, and cingulate cortex, as well as hemispheric asymmetries in STG and cingulate cortex in terms of cortical folding patterns. Moreover, we have also validated the proposed method on a public adult dataset, i. e. , the Human Connectome Project (HCP), and revealed that certain major cortical folding patterns of adults are largely established at term birth.

YNIMG Journal 2019 Journal Article

First-year development of modules and hubs in infant brain functional networks

• Xuyun Wen
• Han Zhang
• Gang Li
• Mingxia Liu
• Weiyan Yin
• Weili Lin
• Jun Zhang
• Dinggang Shen

The human brain develops rapidly in the first postnatal year, in which rewired functional brain networks could shape later behavioral and cognitive performance. Resting-state functional magnetic resonances imaging (rs-fMRI) and complex network analysis have been widely used for characterizing the developmental brain functional connectome. Yet, such studies focusing on the first year of postnatal life are still very limited. Leveraging normally developing longitudinal infant rs-fMRI scans from neonate to one year of age, we investigated how brain functional networks develop at a fine temporal scale (every 3 months). Considering challenges in the infant fMRI-based network analysis, we developed a novel algorithm to construct the robust, temporally consistent and modular structure augmented group-level network based on which functional modules were detected at each age. Our study reveals that the brain functional network is gradually subdivided into an increasing number of functional modules accompanied by the strengthened intra- and inter-modular connectivities. Based on the developing modules, we found connector hubs (the high-centrality regions connecting different modules) emerging and increasing, while provincial hubs (the high-centrality regions connecting regions in the same module) diminishing. Further region-wise longitudinal analysis validates that different hubs have distinct developmental trajectories of the intra- and inter-modular connections suggesting different types of role transitions in network, such as non-hubs to hubs or provincial hubs to connector hubs et al. All findings indicate that functional segregation and integration are both increased in the first year of postnatal life. The module reorganization and hub transition lead to more efficient brain networks, featuring increasingly segregated modular structure and more connector hubs. This study provides the first comprehensive report of the development of functional brain networks at a 3-month interval throughout the first postnatal year of life, which provides essential information to the future neurodevelopmental and developmental disorder studies.

JBHI Journal 2019 Journal Article

Functional Brain Network Estimation With Time Series Self-Scrubbing

• Weikai Li
• Lishan Qiao
• Limei Zhang
• Zhengxia Wang
• Dinggang Shen

Functional brain network (FBN) is becoming an increasingly important measurement for exploring cerebral mechanisms and mining informative biomarkers that assist diagnosis of some neurodegenerative disorders. Despite its effectiveness to discover valuable hidden patterns in the human brain, the estimated FBNs are often heavily influenced by the quality of the observed data (e. g. , blood oxygen level dependent signal series). In practice, a preprocessing pipeline is usually employed for improving data quality. With this in mind, some data points (volumes or time course in the time series) are still not clean enough, due to artifacts including spurious resting-state processes (head movement, mind-wandering). Therefore, not all volumes in the fMRI time series can contribute to the subsequent FBN estimation. To address this issue, we propose a novel FBN estimation method by introducing a latent variable as an indicator of the data quality, and develop an alternating optimization algorithm for jointly scrubbing the data and estimating FBN simultaneously. To further illustrate the effectiveness of the proposed method, we conduct experiments on two public datasets to identify subjects with mild cognitive impairment from normal controls based on the estimated FBNs, and achieve improved accuracies than the baseline methods.

JBHI Journal 2019 Journal Article

Guest Editorial Skin Lesion Image Analysis for Melanoma Detection

• M. Emre Celebi
• Noel Codella
• Allan Halpern
• Dinggang Shen

The papers in this special section focus on the use of image analysis to detect Melanoma. Melanoma is deadliest form of skin cancer, with roughly 91, 000 new cases reported every year in the US and more than 9, 000 deaths. Unlike many other cancer types, the incidence rate of melanoma has been steadily increasing in the past several decades. Early diagnosis is crucial since melanoma can be cured with a simple excision, if detected early. The goals of this special issue are to summarize the state-ofthe- art in the automated analysis of skin lesion images and to provide future directions for this exciting subfield of medical image analysis. The intended audience includes researchers and practicing clinicians, who are increasingly using digital analytic tools.

YNIMG Journal 2019 Journal Article

Multi-task prediction of infant cognitive scores from longitudinal incomplete neuroimaging data

• Ehsan Adeli
• Yu Meng
• Gang Li
• Weili Lin
• Dinggang Shen

Early postnatal brain undergoes a stunning period of development. Over the past few years, research on dynamic infant brain development has received increased attention, exhibiting how important the early stages of a child's life are in terms of brain development. To precisely chart the early brain developmental trajectories, longitudinal studies with data acquired over a long-enough period of infants' early life is essential. However, in practice, missing data from different time point(s) during the data gathering procedure is often inevitable. This leads to incomplete set of longitudinal data, which poses a major challenge for such studies. In this paper, prediction of multiple future cognitive scores with incomplete longitudinal imaging data is modeled into a multi-task machine learning framework. To efficiently learn this model, we account for selection of informative features (i. e. , neuroimaging morphometric measurements for different time points), while preserving the structural information and the interrelation between these multiple cognitive scores. Several experiments are conducted on a carefully acquired in-house dataset, and the results affirm that we can predict the cognitive scores measured at the age of four years old, using the imaging data of earlier time points, as early as 24 months of age, with a reasonable performance (i. e. , root mean square error of 0. 18).

JBHI Journal 2019 Journal Article

Regression Convolutional Neural Network for Automated Pediatric Bone Age Assessment From Hand Radiograph

• Xuhua Ren
• Tingting Li
• Xiujun Yang
• Shuai Wang
• Sahar Ahmad
• Lei Xiang
• Shaun Richard Stone
• Lihong Li

Skeletal bone age assessment is a common clinical practice to investigate endocrinology, and genetic and growth disorders of children. However, clinical interpretation and bone age analyses are time-consuming, labor intensive, and often subject to inter-observer variability. This advocates the need of a fully automated method for bone age assessment. We propose a regression convolutional neural network (CNN) to automatically assess the pediatric bone age from hand radiograph. Our network is specifically trained to place more attention to those bone age related regions in the X-ray images. Specifically, we first adopt the attention module to process all images and generate the coarse/fine attention maps as inputs for the regression network. Then, the regression CNN follows the supervision of the dynamic attention loss during training; thus, it can estimate the bone age of the hard (or “outlier”) images more accurately. The experimental results show that our method achieves an average discrepancy of 5. 2–5. 3 months between clinical and automatic bone age evaluations on two large datasets. In conclusion, we propose a fully automated deep learning solution to process X-ray images of the hand for bone age assessment, with the accuracy comparable to human experts but with much better efficiency.

YNIMG Journal 2019 Journal Article

Resting-state functional MRI studies on infant brains: A decade of gap-filling efforts

• Han Zhang
• Dinggang Shen
• Weili Lin

Resting-state functional MRI (rs-fMRI) is one of the most prevalent brain functional imaging modalities. Previous rs-fMRI studies have mainly focused on adults and elderly subjects. Recently, infant rs-fMRI studies have become an area of active research. After a decade of gap filling studies, many facets of the brain functional development from early infancy to toddler has been uncovered. However, infant rs-fMRI is still in its infancy. The image analysis tools for neonates and young infants can be quite different from those for adults. From data analysis to result interpretation, more questions and issues have been raised, and new hypotheses have been formed. With the anticipated availability of unprecedented high-resolution rs-fMRI and dedicated analysis pipelines from the Baby Connectome Project (BCP), it is important now to revisit previous findings and hypotheses, discuss and comment existing issues and problems, and make a “to-do-list” for the future studies. This review article aims to comprehensively review a decade of the findings, unveiling hidden jewels of the fields of developmental neuroscience and neuroimage computing. Emphases will be given to early infancy, particularly the first few years of life. In this review, an end-to-end summary, from infant rs-fMRI experimental design to data processing, and from the development of individual functional systems to large-scale brain functional networks, is provided. A comprehensive summary of the rs-fMRI findings in developmental patterns is highlighted. Furthermore, an extensive summary of the neurodevelopmental disorders and the effects of other hazardous factors is provided. Finally, future research trends focusing on emerging dynamic functional connectivity and state-of-the-art functional connectome analysis are summarized. In next decade, early infant rs-fMRI and developmental connectome study could be one of the shining research topics.

YNIMG Journal 2019 Journal Article

The UNC/UMN Baby Connectome Project (BCP): An overview of the study design and protocol development

• Brittany R. Howell
• Martin A. Styner
• Wei Gao
• Pew-Thian Yap
• Li Wang
• Kristine Baluyot
• Essa Yacoub
• Geng Chen

The human brain undergoes extensive and dynamic growth during the first years of life. The UNC/UMN Baby Connectome Project (BCP), one of the Lifespan Connectome Projects funded by NIH, is an ongoing study jointly conducted by investigators at the University of North Carolina at Chapel Hill and the University of Minnesota. The primary objective of the BCP is to characterize brain and behavioral development in typically developing infants across the first 5 years of life. The ultimate goals are to chart emerging patterns of structural and functional connectivity during this period, map brain-behavior associations, and establish a foundation from which to further explore trajectories of health and disease. To accomplish these goals, we are combining state of the art MRI acquisition and analysis techniques, including high-resolution structural MRI (T1-and T2-weighted images), diffusion imaging (dMRI), and resting state functional connectivity MRI (rfMRI). While the overall design of the BCP largely is built on the protocol developed by the Lifespan Human Connectome Project (HCP), given the unique age range of the BCP cohort, additional optimization of imaging parameters and consideration of an age appropriate battery of behavioral assessments were needed. Here we provide the overall study protocol, including approaches for subject recruitment, strategies for imaging typically developing children 0–5 years of age without sedation, imaging protocol and optimization, a description of the battery of behavioral assessments, and QA/QC procedures. Combining HCP inspired neuroimaging data with well-established behavioral assessments during this time period will yield an invaluable resource for the scientific community.

YNIMG Journal 2019 Journal Article

Topological correction of infant white matter surfaces using anatomically constrained convolutional neural network

• Liang Sun
• Daoqiang Zhang
• Chunfeng Lian
• Li Wang
• Zhengwang Wu
• WEI SHAO
• Weili Lin
• Dinggang Shen

Reconstruction of accurate cortical surfaces without topological errors (i. e. , handles and holes) from infant brain MR images is very important in early brain development studies. However, infant brain MR images typically suffer extremely low tissue contrast and dynamic imaging appearance patterns. Thus, it is inevitable to have large amounts of topological errors in the segmented infant brain tissue images, which lead to inaccurately reconstructed cortical surfaces with topological errors. To address this issue, inspired by recent advances in deep learning, we propose an anatomically constrained network for topological correction on infant cortical surfaces. Specifically, in our method, we first locate regions of potential topological defects by leveraging a topology-preserving level set method. Then, we propose an anatomically constrained network to correct those candidate voxels in the located regions. Since infant cortical surfaces often contain large and complex handles or holes, it is difficult to completely correct all errors using one-shot correction. Therefore, we further enroll these two steps into an iterative framework to gradually correct large topological errors. To the best of our knowledge, this is the first work to introduce deep learning approach for topological correction of infant cortical surfaces. We compare our method with the state-of-the-art methods on both simulated topological errors and real topological errors in human infant brain MR images. Moreover, we also validate our method on the infant brain MR images of macaques. All experimental results show the superior performance of the proposed method.

YNIMG Journal 2018 Journal Article

3D conditional generative adversarial networks for high-quality PET image estimation at low dose

• Yan Wang
• Biting Yu
• Lei Wang
• Chen Zu
• David S. Lalush
• Weili Lin
• Xi Wu
• Jiliu Zhou

Positron emission tomography (PET) is a widely used imaging modality, providing insight into both the biochemical and physiological processes of human body. Usually, a full dose radioactive tracer is required to obtain high-quality PET images for clinical needs. This inevitably raises concerns about potential health hazards. On the other hand, dose reduction may cause the increased noise in the reconstructed PET images, which impacts the image quality to a certain extent. In this paper, in order to reduce the radiation exposure while maintaining the high quality of PET images, we propose a novel method based on 3D conditional generative adversarial networks (3D c-GANs) to estimate the high-quality full-dose PET images from low-dose ones. Generative adversarial networks (GANs) include a generator network and a discriminator network which are trained simultaneously with the goal of one beating the other. Similar to GANs, in the proposed 3D c-GANs, we condition the model on an input low-dose PET image and generate a corresponding output full-dose PET image. Specifically, to render the same underlying information between the low-dose and full-dose PET images, a 3D U-net-like deep architecture which can combine hierarchical features by using skip connection is designed as the generator network to synthesize the full-dose image. In order to guarantee the synthesized PET image to be close to the real one, we take into account of the estimation error loss in addition to the discriminator feedback to train the generator network. Furthermore, a concatenated 3D c-GANs based progressive refinement scheme is also proposed to further improve the quality of estimated images. Validation was done on a real human brain dataset including both the normal subjects and the subjects diagnosed as mild cognitive impairment (MCI). Experimental results show that our proposed 3D c-GANs method outperforms the benchmark methods and achieves much better performance than the state-of-the-art methods in both qualitative and quantitative measures.

JBHI Journal 2018 Journal Article

Anatomical Landmark Based Deep Feature Representation for MR Images in Brain Disease Diagnosis

• Mingxia Liu
• Jun Zhang
• Dong Nie
• Pew-Thian Yap
• Dinggang Shen

Most automated techniques for brain disease diagnosis utilize hand-crafted (e. g. , voxel-based or region-based) biomarkers from structural magnetic resonance (MR) images as feature representations. However, these hand-crafted features are usually high-dimensional or require regions-of-interest defined by experts. Also, because of possibly heterogeneous property between the hand-crafted features and the subsequent model, existing methods may lead to sub-optimal performances in brain disease diagnosis. In this paper, we propose a landmark-based deep feature learning (LDFL) framework to automatically extract patch-based representation from MRI for automatic diagnosis of Alzheimer's disease. We first identify discriminative anatomical landmarks from MR images in a data-driven manner, and then propose a convolutional neural network for patch-based deep feature learning. We have evaluated the proposed method on subjects from three public datasets, including the Alzheimer's disease neuroimaging initiative (ADNI-1), ADNI-2, and the minimal interval resonance imaging in alzheimer's disease (MIRIAD) dataset. Experimental results of both tasks of brain disease classification and MR image retrieval demonstrate that the proposed LDFL method improves the performance of disease classification and MR image retrieval.

JBHI Journal 2018 Journal Article

Multi-Hypergraph Learning for Incomplete Multimodality Data

• Mingxia Liu
• Yue Gao
• Pew-Thian Yap
• Dinggang Shen

Multi-modality data convey complementary information that can be used to improve the accuracy of prediction models in disease diagnosis. However, effectively integrating multi-modality data remains a challenging problem, especially when the data are incomplete. For instance, more than half of the subjects in the Alzheimer's disease neuroimaging initiative (ADNI) database have no fluorodeoxyglucose positron emission tomography and cerebrospinal fluid data. Currently, there are two commonly used strategies to handle the problem of incomplete data: 1) discard samples having missing features; and 2) impute those missing values via specific techniques. In the first case, a significant amount of useful information is lost and, in the second case, additional noise and artifacts might be introduced into the data. Also, previous studies generally focus on the pairwise relationships among subjects, without considering their underlying complex (e. g. , high-order) relationships. To address these issues, in this paper, we propose a multi-hypergraph learning method for dealing with incomplete multimodality data. Specifically, we first construct multiple hypergraphs to represent the high-order relationships among subjects by dividing them into several groups according to the availability of their data modalities. A hypergraph regularized transductive learning method is then applied to these groups for automatic diagnosis of brain diseases. Extensive evaluation of the proposed method using all subjects in the baseline ADNI database indicates that our method achieves promising results in AD/MCI classification, compared with the state-of-the-art methods.

AAAI Conference 2018 Conference Paper

Multi-Layer Multi-View Classification for Alzheimer’s Disease Diagnosis

• Changqing Zhang
• Ehsan Adeli
• Tao Zhou
• Xiaobo Chen
• Dinggang Shen

In this paper, we propose a novel multi-view learning method for Alzheimer’s Disease (AD) diagnosis, using neuroimaging and genetics data. Generally, there are several major challenges associated with traditional classification methods on multi-source imaging and genetics data. First, the correlation between the extracted imaging features and class labels is generally complex, which often makes the traditional linear models ineffective. Second, medical data may be collected from different sources (i. e. , multiple modalities of neuroimaging data, clinical scores or genetics measurements), therefore, how to effectively exploit the complementarity among multiple views is of great importance. In this paper, we propose a Multi-Layer Multi-View Classification (ML-MVC) approach, which regards the multi-view input as the first layer, and constructs a latent representation to explore the complex correlation between the features and class labels. This captures the high-order complementarity among different views, as we exploit the underlying information with a low-rank tensor regularization. Intrinsically, our formulation elegantly explores the nonlinear correlation together with complementarity among different views, and thus improves the accuracy of classification. Finally, the minimization problem is solved by the Alternating Direction Method of Multipliers (ADMM). Experimental results on Alzheimers Disease Neuroimaging Initiative (ADNI) data sets validate the effectiveness of our proposed method.

JBHI Journal 2017 Journal Article

Alzheimer's Disease Diagnosis Using Landmark-Based Features From Longitudinal Structural MR Images

• Jun Zhang
• Mingxia Liu
• Le An
• Yaozong Gao
• Dinggang Shen

Structural magnetic resonance imaging (MRI) has been proven to be an effective tool for Alzheimer's disease (AD) diagnosis. While conventional MRI-based AD diagnosis typically uses images acquired at a single time point, a longitudinal study is more sensitive in detecting early pathological changes of AD, making it more favorable for accurate diagnosis. In general, there are two challenges faced in MRI-based diagnosis. First, extracting features from structural MR images requires time-consuming nonlinear registration and tissue segmentation, whereas the longitudinal study with involvement of more scans further exacerbates the computational costs. Moreover, the inconsistent longitudinal scans (i. e. , different scanning time points and also the total number of scans) hinder extraction of unified feature representations in longitudinal studies. In this paper, we propose a landmark-based feature extraction method for AD diagnosis using longitudinal structural MR images, which does not require nonlinear registration or tissue segmentation in the application stage and is also robust to inconsistencies among longitudinal scans. Specifically, first, the discriminative landmarks are automatically discovered from the whole brain using training images, and then efficiently localized using a fast landmark detection method for testing images, without the involvement of any nonlinear registration and tissue segmentation; and second, high-level statistical spatial features and contextual longitudinal features are further extracted based on those detected landmarks, which can characterize spatial structural abnormalities and longitudinal landmark variations. Using these spatial and longitudinal features, a linear support vector machine is finally adopted to distinguish AD subjects or mild cognitive impairment (MCI) subjects from healthy controls (HCs). Experimental results on the Alzheimer's Disease Neuroimaging Initiative database demonstrate the superior performance and efficiency of the proposed method, with classification accuracies of 88. 30% for AD versus HC and 79. 02% for MCI versus HC, respectively.

AAAI Conference 2017 Conference Paper

Beyond IID: Learning to Combine Non-IID Metrics for Vision Tasks

• Yinghuan Shi
• Wenbin Li
• Yang Gao
• Longbing Cao
• Dinggang Shen

Metric learning has been widely employed, especially in various computer vision tasks, with the fundamental assumption that all samples (e. g. , regions/superpixels in images/videos) are independent and identically distributed (IID). However, since the samples are usually spatially-connected or temporally-correlated with their physically-connected neighbours, they are not IID (non-IID for short), which cannot be directly handled by existing methods. Thus, we propose to learn and integrate non-IID metrics (NIME). To incorporate the non-IID spatial/temporal relations, instead of directly using non-IID features and metric learning as previous methods, NIME first builds several non-IID representations on original (non-IID) features by various graph kernel functions, and then automatically learns the metric under the best combination of various non-IID representations. NIME is applied to solve two typical computer vision tasks: interactive image segmentation and histology image identification. The results show that learning and integrating non-IID metrics improves the performance, compared to the IID methods. Moreover, our method achieves results comparable or better than that of the state-of-the-arts.

YNIMG Journal 2017 Journal Article

Joint prediction of longitudinal development of cortical surfaces and white matter fibers from neonatal MRI

• Islem Rekik
• Gang Li
• Pew-Thian Yap
• Geng Chen
• Weili Lin
• Dinggang Shen

The human brain can be modeled as multiple interrelated shapes (or a multishape), each for characterizing one aspect of the brain, such as the cortex and white matter pathways. Predicting the developing multishape is a very challenging task due to the contrasting nature of the developmental trajectories of the constituent shapes: smooth for the cortical surface and non-smooth for white matter tracts due to changes such as bifurcation. We recently addressed this problem and proposed an approach for predicting the multishape developmental spatiotemporal trajectories of infant brains based only on neonatal MRI data using a set of geometric, dynamic, and fiber-to-surface connectivity features. In this paper, we propose two key innovations to further improve the prediction of multishape evolution. First, for a more accurate cortical surface prediction, instead of simply relying on one neonatal atlas to guide the prediction of the multishape, we propose to use multiple neonatal atlases to build a spatially heterogeneous atlas using the multidirectional varifold representation. This individualizes the atlas by locally maximizing its similarity to the testing baseline cortical shape for each cortical region, thereby better representing the baseline testing cortical surface, which founds the multishape prediction process. Second, for temporally consistent fiber prediction, we propose to reliably estimate spatiotemporal connectivity features using low-rank tensor completion, thereby capturing the variability and richness of the temporal development of fibers. Experimental results confirm that the proposed variants significantly improve the prediction performance of our original multishape prediction framework for both cortical surfaces and fiber tracts shape at 3, 6, and 9 months of age. Our pioneering model will pave the way for learning how to predict the evolution of anatomical shapes with abnormal changes. Ultimately, devising accurate shape evolution prediction models that can help quantify and predict the severity of a brain disorder as it progresses will be of great aid in individualized treatment planning.

IJCAI Conference 2017 Conference Paper

Predicting Alzheimer's Disease Cognitive Assessment via Robust Low-Rank Structured Sparse Model

• Jie Xu
• Cheng Deng
• Xinbo Gao
• Dinggang Shen
• Heng Huang

Alzheimer's disease (AD) is a neurodegenerative disorder with slow onset, which could result in the deterioration of the duration of persistent neurological dysfunction. How to identify the informative longitudinal phenotypic neuroimaging markers and predict cognitive measures are crucial to recognize AD at early stage. Many existing models related imaging measures to cognitive status using regression models, but they did not take full consideration of the interaction between cognitive scores. In this paper, we propose a robust low-rank structured sparse regression method (RLSR) to address this issue. The proposed model simultaneously selects effective features and learns the underlying structure between cognitive scores by utilizing novel mixed structured sparsity inducing norms and low-rank approximation. In addition, an efficient algorithm is derived to solve the proposed non-smooth objective function with proved convergence. Empirical studies on cognitive data of the ADNI cohort demonstrate the superior performance of the proposed method.

NeurIPS Conference 2017 Conference Paper

Regularized Modal Regression with Applications in Cognitive Impairment Prediction

• Xiaoqian Wang
• Hong Chen
• Weidong Cai
• Dinggang Shen
• Heng Huang

Linear regression models have been successfully used to function estimation and model selection in high-dimensional data analysis. However, most existing methods are built on least squares with the mean square error (MSE) criterion, which are sensitive to outliers and their performance may be degraded for heavy-tailed noise. In this paper, we go beyond this criterion by investigating the regularized modal regression from a statistical learning viewpoint. A new regularized modal regression model is proposed for estimation and variable selection, which is robust to outliers, heavy-tailed noise, and skewed noise. On the theoretical side, we establish the approximation estimate for learning the conditional mode function, the sparsity analysis for variable selection, and the robustness characterization. On the application side, we applied our model to successfully improve the cognitive impairment prediction using the Alzheimer’s Disease Neuroimaging Initiative (ADNI) cohort data.

YNICL Journal 2016 Journal Article

Automated quantification of cerebral edema following hemispheric infarction: Application of a machine-learning algorithm to evaluate CSF shifts on serial head CTs

• Yasheng Chen
• Rajat Dhar
• Laura Heitsch
• Andria Ford
• Israel Fernandez-Cadenas
• Caty Carrera
• Joan Montaner
• Weili Lin

Although cerebral edema is a major cause of death and deterioration following hemispheric stroke, there remains no validated biomarker that captures the full spectrum of this critical complication. We recently demonstrated that reduction in intracranial cerebrospinal fluid (CSF) volume (∆CSF) on serial computed tomography (CT) scans provides an accurate measure of cerebral edema severity, which may aid in early triaging of stroke patients for craniectomy. However, application of such a volumetric approach would be too cumbersome to perform manually on serial scans in a real-world setting. We developed and validated an automated technique for CSF segmentation via integration of random forest (RF) based machine learning with geodesic active contour (GAC) segmentation. The proposed RF+GAC approach was compared to conventional Hounsfield Unit (HU) thresholding and RF segmentation methods using Dice similarity coefficient (DSC) and the correlation of volumetric measurements, with manual delineation serving as the ground truth. CSF spaces were outlined on scans performed at baseline (<6h after stroke onset) and early follow-up (FU) (closest to 24h) in 38 acute ischemic stroke patients. RF performed significantly better than optimized HU thresholding (p<10−4 in baseline and p<10−5 in FU) and RF+GAC performed significantly better than RF (p<10−3 in baseline and p<10−5 in FU). Pearson correlation coefficients between the automatically detected ∆CSF and the ground truth were r =0. 178 (p=0. 285), r =0. 876 (p<10−6) and r =0. 879 (p<10−6) for thresholding, RF and RF+GAC, respectively, with a slope closer to the line of identity in RF+GAC. When we applied the algorithm trained from images of one stroke center to segment CTs from another center, similar findings held. In conclusion, we have developed and validated an accurate automated approach to segment CSF and calculate its shifts on serial CT scans. This algorithm will allow us to efficiently and accurately measure the evolution of cerebral edema in future studies including large multi-site patient populations.

YNIMG Journal 2016 Journal Article

Estimating functional brain networks by incorporating a modularity prior

• Lishan Qiao
• Han Zhang
• Minjeong Kim
• Shenghua Teng
• Limei Zhang
• Dinggang Shen

Functional brain network analysis has become one principled way of revealing informative organization architectures in healthy brains, and providing sensitive biomarkers for diagnosis of neurological disorders. Prior to any post hoc analysis, however, a natural issue is how to construct “ideal” brain networks given, for example, a set of functional magnetic resonance imaging (fMRI) time series associated with different brain regions. Although many methods have been developed, it is currently still an open field to estimate biologically meaningful and statistically robust brain networks due to our limited understanding of the human brain as well as complex noises in the observed data. Motivated by the fact that the brain is organized with modular structures, in this paper, we propose a novel functional brain network modeling scheme by encoding a modularity prior under a matrix-regularized network learning framework, and further formulate it as a sparse low-rank graph learning problem, which can be solved by an efficient optimization algorithm. Then, we apply the learned brain networks to identify patients with mild cognitive impairment (MCI) from normal controls. We achieved 89. 01% classification accuracy even with a simple feature selection and classification pipeline, which significantly outperforms the conventional brain network construction methods. Moreover, we further explore brain network features that contributed to MCI identification, and discovered potential biomarkers for personalized diagnosis.

YNIMG Journal 2016 Journal Article

Joint feature-sample selection and robust diagnosis of Parkinson's disease from MRI data

• Ehsan Adeli
• Feng Shi
• Le An
• Chong-Yaw Wee
• Guorong Wu
• Tao Wang
• Dinggang Shen

Parkinson's disease (PD) is an overwhelming neurodegenerative disorder caused by deterioration of a neurotransmitter, known as dopamine. Lack of this chemical messenger impairs several brain regions and yields various motor and non-motor symptoms. Incidence of PD is predicted to double in the next two decades, which urges more research to focus on its early diagnosis and treatment. In this paper, we propose an approach to diagnose PD using magnetic resonance imaging (MRI) data. Specifically, we first introduce a joint feature-sample selection (JFSS) method for selecting an optimal subset of samples and features, to learn a reliable diagnosis model. The proposed JFSS model effectively discards poor samples and irrelevant features. As a result, the selected features play an important role in PD characterization, which will help identify the most relevant and critical imaging biomarkers for PD. Then, a robust classification framework is proposed to simultaneously de-noise the selected subset of features and samples, and learn a classification model. Our model can also de-noise testing samples based on the cleaned training data. Unlike many previous works that perform de-noising in an unsupervised manner, we perform supervised de-noising for both training and testing data, thus boosting the diagnostic accuracy. Experimental results on both synthetic and publicly available PD datasets show promising results. To evaluate the proposed method, we use the popular Parkinson's progression markers initiative (PPMI) database. Our results indicate that the proposed method can differentiate between PD and normal control (NC), and outperforms the competing methods by a relatively large margin. It is noteworthy to mention that our proposed framework can also be used for diagnosis of other brain disorders. To show this, we have also conducted experiments on the widely-used ADNI database. The obtained results indicate that our proposed method can identify the imaging biomarkers and diagnose the disease with favorable accuracies compared to the baseline methods.

YNIMG Journal 2016 Journal Article

Multidirectional and Topography-based Dynamic-scale Varifold Representations with Application to Matching Developing Cortical Surfaces

• Islem Rekik
• Gang Li
• Weili Lin
• Dinggang Shen

The human cerebral cortex is marked by great complexity as well as substantial dynamic changes during early postnatal development. To obtain a fairly comprehensive picture of its age-induced and/or disorder-related cortical changes, one needs to match cortical surfaces to one another, while maximizing their anatomical alignment. Methods that geodesically shoot surfaces into one another as currents (a distribution of oriented normals) and varifolds (a distribution of non-oriented normals) provide an elegant Riemannian framework for generic surface matching and reliable statistical analysis. However, both conventional current and varifold matching methods have two key limitations. First, they only use the normals of the surface to measure its geometry and guide the warping process, which overlooks the importance of the orientations of the inherently convoluted cortical sulcal and gyral folds. Second, the ‘conversion’ of a surface into a current or a varifold operates at a fixed scale under which geometric surface details will be neglected, which ignores the dynamic scales of cortical foldings. To overcome these limitations and improve varifold-based cortical surface registration, we propose two different strategies. The first strategy decomposes each cortical surface into its normal and tangent varifold representations, by integrating principal curvature direction field into the varifold matching framework, thus providing rich information of the orientation of cortical folding and better characterization of the complex cortical geometry. The second strategy explores the informative cortical geometric features to perform a dynamic-scale measurement of the cortical surface that depends on the local surface topography (e. g. , principal curvature), thereby we introduce the concept of a topography-based dynamic-scale varifold. We tested the proposed varifold variants for registering 12 pairs of dynamically developing cortical surfaces from 0 to 6 months of age. Both variants improved the matching accuracy in terms of closeness to the target surface and the goodness of alignment with regional anatomical boundaries, when compared with three state-of-the-art methods: (1) diffeomorphic spectral matching, (2) conventional current-based surface matching, and (3) conventional varifold-based surface matching.

YNIMG Journal 2016 Journal Article

Segmentation of perivascular spaces in 7 T MR image using auto-context model with orientation-normalized features

• Sang Hyun Park
• Xiaopeng Zong
• Yaozong Gao
• Weili Lin
• Dinggang Shen

Quantitative study of perivascular spaces (PVSs) in brain magnetic resonance (MR) images is important for understanding the brain lymphatic system and its relationship with neurological diseases. One of the major challenges is the accurate extraction of PVSs that have very thin tubular structures with various directions in three-dimensional (3D) MR images. In this paper, we propose a learning-based PVS segmentation method to address this challenge. Specifically, we first determine a region of interest (ROI) by using the anatomical brain structure and the vesselness information derived from eigenvalues of image derivatives. Then, in the ROI, we extract a number of randomized Haar features which are normalized with respect to the principal directions of the underlying image derivatives. The classifier is trained by the random forest model that can effectively learn both discriminative features and classifier parameters to maximize the information gain. Finally, a sequential learning strategy is used to further enforce various contextual patterns around the thin tubular structures into the classifier. For evaluation, we apply our proposed method to the 7T brain MR images scanned from 17 healthy subjects aged from 25 to 37. The performance is measured by voxel-wise segmentation accuracy, cluster-wise classification accuracy, and similarity of geometric properties, such as volume, length, and diameter distributions between the predicted and the true PVSs. Moreover, the accuracies are also evaluated on the simulation images with motion artifacts and lacunes to demonstrate the potential of our method in segmenting PVSs from elderly and patient populations. The experimental results show that our proposed method outperforms all existing PVS segmentation methods.

YNIMG Journal 2016 Journal Article

State-space model with deep learning for functional dynamics estimation in resting-state fMRI

• Heung-Il Suk
• Chong-Yaw Wee
• Seong-Whan Lee
• Dinggang Shen

Studies on resting-state functional Magnetic Resonance Imaging (rs-fMRI) have shown that different brain regions still actively interact with each other while a subject is at rest, and such functional interaction is not stationary but changes over time. In terms of a large-scale brain network, in this paper, we focus on time-varying patterns of functional networks, i. e. , functional dynamics, inherent in rs-fMRI, which is one of the emerging issues along with the network modelling. Specifically, we propose a novel methodological architecture that combines deep learning and state-space modelling, and apply it to rs-fMRI based Mild Cognitive Impairment (MCI) diagnosis. We first devise a Deep Auto-Encoder (DAE) to discover hierarchical non-linear functional relations among regions, by which we transform the regional features into an embedding space, whose bases are complex functional networks. Given the embedded functional features, we then use a Hidden Markov Model (HMM) to estimate dynamic characteristics of functional networks inherent in rs-fMRI via internal states, which are unobservable but can be inferred from observations statistically. By building a generative model with an HMM, we estimate the likelihood of the input features of rs-fMRI as belonging to the corresponding status, i. e. , MCI or normal healthy control, based on which we identify the clinical label of a testing subject. In order to validate the effectiveness of the proposed method, we performed experiments on two different datasets and compared with state-of-the-art methods in the literature. We also analyzed the functional networks learned by DAE, estimated the functional connectivities by decoding hidden states in HMM, and investigated the estimated functional connectivities by means of a graph-theoretic approach.

YNIMG Journal 2016 Journal Article

Visualization of perivascular spaces in the human brain at 7 T: sequence optimization and morphology characterization

• Xiaopeng Zong
• Sang Hyun Park
• Dinggang Shen
• Weili Lin

Noninvasive imaging of perivascular spaces (PVSs) may provide useful insights into their role in normal brain physiology and diseases. Fast MRI sequences with sub-millimeter spatial resolutions and high contrast-to-noise ratio (CNR) are required for accurate delineation of PVS in human. To achieve the optimal condition for PVS imaging at 7T, we carried out detailed simulation and experimental studies to characterize the dependence of CNR on imaging sequences (T1 versus T2-weighted) and sequence parameters. In addition, PVSs were segmented semi-automatically, which revealed much larger numbers of PVSs in young healthy subjects (age 21–37years) than previously reported. To the best of our knowledge, our study provides, for the first time, detailed length, volume, and diameter distributions of PVS in the white matter and subcortical nuclei, which can serve as a reference for future studies of PVS abnormalities under diseased conditions.

JBHI Journal 2015 Journal Article

A Robust Deep Model for Improved Classification of AD/MCI Patients

• Feng Li
• Loc Tran
• Kim-Han Thung
• Shuiwang Ji
• Dinggang Shen
• Jiang Li

Accurate classification of Alzheimer's disease (AD) and its prodromal stage, mild cognitive impairment (MCI), plays a critical role in possibly preventing progression of memory impairment and improving quality of life for AD patients. Among many research tasks, it is of a particular interest to identify noninvasive imaging biomarkers for AD diagnosis. In this paper, we present a robust deep learning system to identify different progression stages of AD patients based on MRI and PET scans. We utilized the dropout technique to improve classical deep learning by preventing its weight coadaptation, which is a typical cause of overfitting in deep learning. In addition, we incorporated stability selection, an adaptive learning factor, and a multitask learning strategy into the deep learning framework. We applied the proposed method to the ADNI dataset, and conducted experiments for AD and MCI conversion diagnosis. Experimental results showed that the dropout technique is very effective in AD diagnosis, improving the classification accuracies by 5. 9% on average as compared to the classical deep learning methods.

YNIMG Journal 2015 Journal Article

Deep convolutional neural networks for multi-modality isointense infant brain image segmentation

• Wenlu Zhang
• Rongjian Li
• Houtao Deng
• Li Wang
• Weili Lin
• Shuiwang Ji
• Dinggang Shen

The segmentation of infant brain tissue images into white matter (WM), gray matter (GM), and cerebrospinal fluid (CSF) plays an important role in studying early brain development in health and disease. In the isointense stage (approximately 6–8 months of age), WM and GM exhibit similar levels of intensity in both T1 and T2 MR images, making the tissue segmentation very challenging. Only a small number of existing methods have been designed for tissue segmentation in this isointense stage; however, they only used a single T1 or T2 images, or the combination of T1 and T2 images. In this paper, we propose to use deep convolutional neural networks (CNNs) for segmenting isointense stage brain tissues using multi-modality MR images. CNNs are a type of deep models in which trainable filters and local neighborhood pooling operations are applied alternatingly on the raw input images, resulting in a hierarchy of increasingly complex features. Specifically, we used multi-modality information from T1, T2, and fractional anisotropy (FA) images as inputs and then generated the segmentation maps as outputs. The multiple intermediate layers applied convolution, pooling, normalization, and other operations to capture the highly nonlinear mappings between inputs and outputs. We compared the performance of our approach with that of the commonly used segmentation methods on a set of manually segmented isointense stage brain images. Results showed that our proposed model significantly outperformed prior methods on infant brain tissue segmentation. In addition, our results indicated that integration of multi-modality images led to significant performance improvement.

JBHI Journal 2015 Journal Article

Editorial: Machine Learning and Data Mining in Medical Imaging

• Dinggang Shen
• Daoqiang Zhang
• Alastair Young
• Bahram Parvin

Presents the introductory editorial for this issue of the publication.

YNIMG Journal 2015 Journal Article

Evaluation of machine learning algorithms for treatment outcome prediction in patients with epilepsy based on structural connectome data

• Brent C. Munsell
• Chong-Yaw Wee
• Simon S. Keller
• Bernd Weber
• Christian Elger
• Laura Angelica Tomaz da Silva
• Travis Nesland
• Martin Styner

The objective of this study is to evaluate machine learning algorithms aimed at predicting surgical treatment outcomes in groups of patients with temporal lobe epilepsy (TLE) using only the structural brain connectome. Specifically, the brain connectome is reconstructed using white matter fiber tracts from presurgical diffusion tensor imaging. To achieve our objective, a two-stage connectome-based prediction framework is developed that gradually selects a small number of abnormal network connections that contribute to the surgical treatment outcome, and in each stage a linear kernel operation is used to further improve the accuracy of the learned classifier. Using a 10-fold cross validation strategy, the first stage in the connectome-based framework is able to separate patients with TLE from normal controls with 80% accuracy, and second stage in the connectome-based framework is able to correctly predict the surgical treatment outcome of patients with TLE with 70% accuracy. Compared to existing state-of-the-art methods that use VBM data, the proposed two-stage connectome-based prediction framework is a suitable alternative with comparable prediction performance. Our results additionally show that machine learning algorithms that exclusively use structural connectome data can predict treatment outcomes in epilepsy with similar accuracy compared with “expert-based” clinical decision. In summary, using the unprecedented information provided in the brain connectome, machine learning algorithms may uncover pathological changes in brain network organization and improve outcome forecasting in the context of epilepsy.

YNIMG Journal 2015 Journal Article

Hierarchical multi-atlas label fusion with multi-scale feature representation and label-specific patch partition

• Guorong Wu
• Minjeong Kim
• Gerard Sanroma
• Qian Wang
• Brent C. Munsell
• Dinggang Shen

Multi-atlas patch-based label fusion methods have been successfully used to improve segmentation accuracy in many important medical image analysis applications. In general, to achieve label fusion a single target image is first registered to several atlas images. After registration a label is assigned to each target point in the target image by determining the similarity between the underlying target image patch (centered at the target point) and the aligned image patch in each atlas image. To achieve the highest level of accuracy during the label fusion process it's critical for the chosen patch similarity measurement to accurately capture the tissue/shape appearance of the anatomical structure. One major limitation of existing state-of-the-art label fusion methods is that they often apply a fixed size image patch throughout the entire label fusion procedure. Doing so may severely affect the fidelity of the patch similarity measurement, which in turn may not adequately capture complex tissue appearance patterns expressed by the anatomical structure. To address this limitation, we advance state-of-the-art by adding three new label fusion contributions: First, each image patch is now characterized by a multi-scale feature representation that encodes both local and semi-local image information. Doing so will increase the accuracy of the patch-based similarity measurement. Second, to limit the possibility of the patch-based similarity measurement being wrongly guided by the presence of multiple anatomical structures in the same image patch, each atlas image patch is further partitioned into a set of label-specific partial image patches according to the existing labels. Since image information has now been semantically divided into different patterns, these new label-specific atlas patches make the label fusion process more specific and flexible. Lastly, in order to correct target points that are mislabeled during label fusion, a hierarchical approach is used to improve the label fusion results. In particular, a coarse-to-fine iterative label fusion approach is used that gradually reduces the patch size. To evaluate the accuracy of our label fusion approach, the proposed method was used to segment the hippocampus in the ADNI dataset and 7. 0T MR images, sub-cortical regions in LONI LBPA40 dataset, mid-brain regions in SATA dataset from MICCAI 2013 segmentation challenge, and a set of key internal gray matter structures in IXI dataset. In all experiments, the segmentation results of the proposed hierarchical label fusion method with multi-scale feature representations and label-specific atlas patches are more accurate than several well-known state-of-the-art label fusion methods.

YNIMG Journal 2015 Journal Article

Improved image registration by sparse patch-based deformation estimation

• Minjeong Kim
• Guorong Wu
• Qian Wang
• Seong-Whan Lee
• Dinggang Shen

Despite intensive efforts for decades, deformable image registration is still a challenging problem due to the potential large anatomical differences across individual images, which limits the registration performance. Fortunately, this issue could be alleviated if a good initial deformation can be provided for the two images under registration, which are often termed as the moving subject and the fixed template, respectively. In this work, we present a novel patch-based initial deformation prediction framework for improving the performance of existing registration algorithms. Our main idea is to estimate the initial deformation between subject and template in a patch-wise fashion by using the sparse representation technique. We argue that two image patches should follow the same deformation toward the template image if their patch-wise appearance patterns are similar. To this end, our framework consists of two stages, i. e. , the training stage and the application stage. In the training stage, we register all training images to the pre-selected template, such that the deformation of each training image with respect to the template is known. In the application stage, we apply the following four steps to efficiently calculate the initial deformation field for the new test subject: (1) We pick a small number of key points in the distinctive regions of the test subject; (2) for each key point, we extract a local patch and form a coupled appearance-deformation dictionary from training images where each dictionary atom consists of the image intensity patch as well as their respective local deformations; (3) a small set of training image patches in the coupled dictionary are selected to represent the image patch of each subject key point by sparse representation. Then, we can predict the initial deformation for each subject key point by propagating the pre-estimated deformations on the selected training patches with the same sparse representation coefficients; and (4) we employ thin-plate splines (TPS) to interpolate a dense initial deformation field by considering all key points as the control points. Thus, the conventional image registration problem becomes much easier in the sense that we only need to compute the remaining small deformation for completing the registration of the subject to the template. Experimental results on both simulated and real data show that the registration performance can be significantly improved after integrating our patch-based deformation prediction framework into the existing registration algorithms.

YNIMG Journal 2015 Journal Article

LINKS: Learning-based multi-source IntegratioN frameworK for Segmentation of infant brain images

• Li Wang
• Yaozong Gao
• Feng Shi
• Gang Li
• John H. Gilmore
• Weili Lin
• Dinggang Shen

Segmentation of infant brain MR images is challenging due to insufficient image quality, severe partial volume effect, and ongoing maturation and myelination processes. In the first year of life, the image contrast between white and gray matters of the infant brain undergoes dramatic changes. In particular, the image contrast is inverted around 6–8months of age, and the white and gray matter tissues are isointense in both T1- and T2-weighted MR images and thus exhibit the extremely low tissue contrast, which poses significant challenges for automated segmentation. Most previous studies used multi-atlas label fusion strategy, which has the limitation of equally treating the different available image modalities and is often computationally expensive. To cope with these limitations, in this paper, we propose a novel learning-based multi-source integration framework for segmentation of infant brain images. Specifically, we employ the random forest technique to effectively integrate features from multi-source images together for tissue segmentation. Here, the multi-source images include initially only the multi-modality (T1, T2 and FA) images and later also the iteratively estimated and refined tissue probability maps of gray matter, white matter, and cerebrospinal fluid. Experimental results on 119 infants show that the proposed method achieves better performance than other state-of-the-art automated segmentation methods. Further validation was performed on the MICCAI grand challenge and the proposed method was ranked top among all competing methods. Moreover, to alleviate the possible anatomical errors, our method can also be combined with an anatomically-constrained multi-atlas labeling approach for further improving the segmentation accuracy.

NeurIPS Conference 2015 Conference Paper

Robust Feature-Sample Linear Discriminant Analysis for Brain Disorders Diagnosis

• Ehsan Adeli-Mosabbeb
• Kim-Han Thung
• Le An
• Feng Shi
• Dinggang Shen

A wide spectrum of discriminative methods is increasingly used in diverse applications for classification or regression tasks. However, many existing discriminative methods assume that the input data is nearly noise-free, which limits their applications to solve real-world problems. Particularly for disease diagnosis, the data acquired by the neuroimaging devices are always prone to different sources of noise. Robust discriminative models are somewhat scarce and only a few attempts have been made to make them robust against noise or outliers. These methods focus on detecting either the sample-outliers or feature-noises. Moreover, they usually use unsupervised de-noising procedures, or separately de-noise the training and the testing data. All these factors may induce biases in the learning process, and thus limit its performance. In this paper, we propose a classification method based on the least-squares formulation of linear discriminant analysis, which simultaneously detects the sample-outliers and feature-noises. The proposed method operates under a semi-supervised setting, in which both labeled training and unlabeled testing data are incorporated to form the intrinsic geometry of the sample space. Therefore, the violating samples or feature values are identified as sample-outliers or feature-noises, respectively. We test our algorithm on one synthetic and two brain neurodegenerative databases (particularly for Parkinson's disease and Alzheimer's disease). The results demonstrate that our method outperforms all baseline and state-of-the-art methods, in terms of both accuracy and the area under the ROC curve.

YNIMG Journal 2014 Journal Article

A novel matrix-similarity based loss function for joint regression and classification in AD diagnosis

• Xiaofeng Zhu
• Heung-Il Suk
• Dinggang Shen

Recent studies on AD/MCI diagnosis have shown that the tasks of identifying brain disease and predicting clinical scores are highly related to each other. Furthermore, it has been shown that feature selection with a manifold learning or a sparse model can handle the problems of high feature dimensionality and small sample size. However, the tasks of clinical score regression and clinical label classification were often conducted separately in the previous studies. Regarding the feature selection, to our best knowledge, most of the previous work considered a loss function defined as an element-wise difference between the target values and the predicted ones. In this paper, we consider the problems of joint regression and classification for AD/MCI diagnosis and propose a novel matrix-similarity based loss function that uses high-level information inherent in the target response matrix and imposes the information to be preserved in the predicted response matrix. The newly devised loss function is combined with a group lasso method for joint feature selection across tasks, i. e. , predictions of clinical scores and a class label. In order to validate the effectiveness of the proposed method, we conducted experiments on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset, and showed that the newly devised loss function helped enhance the performances of both clinical score prediction and disease status identification, outperforming the state-of-the-art methods.

YNIMG Journal 2014 Journal Article

Fiber-driven resolution enhancement of diffusion-weighted images

• Pew-Thian Yap
• Hongyu An
• Yasheng Chen
• Dinggang Shen

Diffusion-weighted imaging (DWI), while giving rich information about brain circuitry, is often limited by insufficient spatial resolution and low signal-to-noise ratio (SNR). This paper describes an algorithm that will increase the resolution of DW images beyond the scan resolution, allowing for a closer investigation of fiber structures and more accurate assessment of brain connectivity. The algorithm is capable of generating a dense vector-valued field, consisting of diffusion data associated with the full set of diffusion-sensitizing gradients. The fundamental premise is that, to best preserve information, interpolation should always be performed along axonal fibers. To achieve this, at each spatial location, we probe neighboring voxels in various directions to gather diffusion information for data interpolation. Based on the fiber orientation distribution function (ODF), directions that are more likely to be traversed by fibers will be given greater weights during interpolation and vice versa. This ensures that data interpolation is only contributed by diffusion data coming from fibers that are aligned with a specific direction. This approach respects local fiber structures and prevents blurring resulting from averaging of data from significantly misaligned fibers. Evaluations suggest that this algorithm yields results with significantly less blocking artifacts, greater smoothness in anatomical structures, and markedly improved structural visibility.

YNIMG Journal 2014 Journal Article

Hierarchical feature representation and multimodal fusion with deep learning for AD/MCI diagnosis

• Heung-Il Suk
• Seong-Whan Lee
• Dinggang Shen

For the last decade, it has been shown that neuroimaging can be a potential tool for the diagnosis of Alzheimer's Disease (AD) and its prodromal stage, Mild Cognitive Impairment (MCI), and also fusion of different modalities can further provide the complementary information to enhance diagnostic accuracy. Here, we focus on the problems of both feature representation and fusion of multimodal information from Magnetic Resonance Imaging (MRI) and Positron Emission Tomography (PET). To our best knowledge, the previous methods in the literature mostly used hand-crafted features such as cortical thickness, gray matter densities from MRI, or voxel intensities from PET, and then combined these multimodal features by simply concatenating into a long vector or transforming into a higher-dimensional kernel space. In this paper, we propose a novel method for a high-level latent and shared feature representation from neuroimaging modalities via deep learning. Specifically, we use Deep Boltzmann Machine (DBM) 2 2 Although it is clear from the context that the acronym DBM denotes “Deep Boltzmann Machine” in this paper, we would clearly indicate that DBM here is not related to “Deformation Based Morphometry”. , a deep network with a restricted Boltzmann machine as a building block, to find a latent hierarchical feature representation from a 3D patch, and then devise a systematic method for a joint feature representation from the paired patches of MRI and PET with a multimodal DBM. To validate the effectiveness of the proposed method, we performed experiments on ADNI dataset and compared with the state-of-the-art methods. In three binary classification problems of AD vs. healthy Normal Control (NC), MCI vs. NC, and MCI converter vs. MCI non-converter, we obtained the maximal accuracies of 95. 35%, 85. 67%, and 74. 58%, respectively, outperforming the competing methods. By visual inspection of the trained model, we observed that the proposed method could hierarchically discover the complex latent patterns inherent in both MRI and PET.

YNIMG Journal 2014 Journal Article

Hierarchical unbiased graph shrinkage (HUGS): A novel groupwise registration for large data set

• Shihui Ying
• Guorong Wu
• Qian Wang
• Dinggang Shen

Normalizing all images in a large data set into a common space is a key step in many clinical and research studies, e. g. , for brain development, maturation, and aging. Recently, groupwise registration has been developed for simultaneous alignment of all images without selecting a particular image as template, thus potentially avoiding bias in the registration. However, most conventional groupwise registration methods do not explore the data distribution during the image registration. Thus, their performance could be affected by large inter-subject variations in the data set under registration. To solve this potential issue, we propose to use a graph to model the distribution of all image data sitting on the image manifold, with each node representing an image and each edge representing the geodesic pathway between two nodes (or images). Then, the procedure of warping all images to their population center turns to the dynamic shrinking of the graph nodes along their graph edges until all graph nodes become close to each other. Thus, the topology of image distribution on the image manifold is always preserved during the groupwise registration. More importantly, by modeling the distribution of all images via a graph, we can potentially reduce registration error since every time each image is warped only according to its nearby images with similar structures in the graph. We have evaluated our proposed groupwise registration method on both infant and adult data sets, by also comparing with the conventional group-mean based registration and the ABSORB methods. All experimental results show that our proposed method can achieve better performance in terms of registration accuracy and robustness.

YNIMG Journal 2014 Journal Article

Integration of sparse multi-modality representation and anatomical constraint for isointense infant brain MR image segmentation

• Li Wang
• Feng Shi
• Yaozong Gao
• Gang Li
• John H. Gilmore
• Weili Lin
• Dinggang Shen

Segmentation of infant brain MR images is challenging due to poor spatial resolution, severe partial volume effect, and the ongoing maturation and myelination processes. During the first year of life, the brain image contrast between white and gray matters undergoes dramatic changes. In particular, the image contrast inverses around 6–8months of age, where the white and gray matter tissues are isointense in T1 and T2 weighted images and hence exhibit the extremely low tissue contrast, posing significant challenges for automated segmentation. In this paper, we propose a general framework that adopts sparse representation to fuse the multi-modality image information and further incorporate the anatomical constraints for brain tissue segmentation. Specifically, we first derive an initial segmentation from a library of aligned images with ground-truth segmentations by using sparse representation in a patch-based fashion for the multi-modality T1, T2 and FA images. The segmentation result is further iteratively refined by integration of the anatomical constraint. The proposed method was evaluated on 22 infant brain MR images acquired at around 6months of age by using a leave-one-out cross-validation, as well as other 10 unseen testing subjects. Our method achieved a high accuracy for the Dice ratios that measure the volume overlap between automated and manual segmentations, i. e. , 0. 889±0. 008 for white matter and 0. 870±0. 006 for gray matter.

YNIMG Journal 2014 Journal Article

Inter-modality relationship constrained multi-modality multi-task feature selection for Alzheimer's Disease and mild cognitive impairment identification

• Feng Liu
• Chong-Yaw Wee
• Huafu Chen
• Dinggang Shen

Previous studies have demonstrated that the use of integrated information from multi-modalities could significantly improve diagnosis of Alzheimer's Disease (AD). However, feature selection, which is one of the most important steps in classification, is typically performed separately for each modality, which ignores the potentially strong inter-modality relationship within each subject. Recent emergence of multi-task learning approach makes the joint feature selection from different modalities possible. However, joint feature selection may unfortunately overlook different yet complementary information conveyed by different modalities. We propose a novel multi-task feature selection method to preserve the complementary inter-modality information. Specifically, we treat feature selection from each modality as a separate task and further impose a constraint for preserving the inter-modality relationship, besides separately enforcing the sparseness of the selected features from each modality. After feature selection, a multi-kernel support vector machine (SVM) is further used to integrate the selected features from each modality for classification. Our method is evaluated using the baseline PET and MRI images of subjects obtained from the Alzheimer's Disease Neuroimaging Initiative (ADNI) database. Our method achieves a good performance, with an accuracy of 94. 37% and an area under the ROC curve (AUC) of 0. 9724 for AD identification, and also an accuracy of 78. 80% and an AUC of 0. 8284 for mild cognitive impairment (MCI) identification. Moreover, the proposed method achieves an accuracy of 67. 83% and an AUC of 0. 6957 for separating between MCI converters and MCI non-converters (to AD). These performances demonstrate the superiority of the proposed method over the state-of-the-art classification methods.

YNIMG Journal 2014 Journal Article

Measuring the dynamic longitudinal cortex development in infants by reconstruction of temporally consistent cortical surfaces

• Gang Li
• Jingxin Nie
• Li Wang
• Feng Shi
• John H. Gilmore
• Weili Lin
• Dinggang Shen

Quantitative measurement of the dynamic longitudinal cortex development during early postnatal stages is of great importance to understand the early cortical structural and functional development. Conventional methods usually reconstruct the cortical surfaces of longitudinal images from the same subject independently, which often generate longitudinally-inconsistent cortical surfaces and thus lead to inaccurate measurement of cortical changes, especially for vertex-wise mapping of cortical development. This paper aims to address this problem by presenting a method to reconstruct temporally-consistent cortical surfaces from longitudinal infant brain MR images, for accurate and consistent measurement of the dynamic cortex development in infants. Specifically, the longitudinal development of the inner cortical surface is first modeled by a deformable growth sheet with elasto-plasticity property to establish longitudinally smooth correspondences of the inner cortical surfaces. Then, the modeled longitudinal inner cortical surfaces are jointly deformed to locate both inner and outer cortical surfaces with a spatial–temporal deformable surface method. The method has been applied to 13 healthy infants, each with 6 serial MR scans acquired at 2weeks, 3months, 6months, 9months, 12months and 18months of age. Experimental results showed that our method with the incorporated longitudinal constraints can reconstruct the longitudinally-dynamic cortical surfaces from serial infant MR images more consistently and accurately than the previously published methods. By using our method, for the first time, we can characterize the vertex-wise longitudinal cortical thickness development trajectory at multiple time points in the first 18months of life. Specifically, we found the highly age-related and regionally-heterogeneous developmental trajectories of the cortical thickness during this period, with the cortical thickness increased most from 3 to 6months (16. 2%) and least from 9 to 12months (less than 0. 1%). Specifically, the central sulcus only underwent significant increase of cortical thickness from 6 to 9months and the occipital cortex underwent significant increase from 0 to 9months, while the frontal, temporal and parietal cortices grew continuously in this first 18months of life. The adult-like spatial patterns of cortical thickness were generally present at 18months of age. These results provided detailed insights into the dynamic trajectory of the cortical thickness development in infants.

YNIMG Journal 2014 Journal Article

Neurodegenerative disease diagnosis using incomplete multi-modality data via matrix shrinkage and completion

• Kim-Han Thung
• Chong-Yaw Wee
• Pew-Thian Yap
• Dinggang Shen

In this work, we are interested in predicting the diagnostic statuses of potentially neurodegenerated patients using feature values derived from multi-modality neuroimaging data and biological data, which might be incomplete. Collecting the feature values into a matrix, with each row containing a feature vector of a sample, we propose a framework to predict the corresponding associated multiple target outputs (e. g. , diagnosis label and clinical scores) from this feature matrix by performing matrix shrinkage following matrix completion. Specifically, we first combine the feature and target output matrices into a large matrix and then partition this large incomplete matrix into smaller submatrices, each consisting of samples with complete feature values (corresponding to a certain combination of modalities) and target outputs. Treating each target output as the outcome of a prediction task, we apply a 2-step multi-task learning algorithm to select the most discriminative features and samples in each submatrix. Features and samples that are not selected in any of the submatrices are discarded, resulting in a shrunk version of the original large matrix. The missing feature values and unknown target outputs of the shrunk matrix is then completed simultaneously. Experimental results using the ADNI dataset indicate that our proposed framework achieves higher classification accuracy at a greater speed when compared with conventional imputation-based classification methods and also yields competitive performance when compared with the state-of-the-art methods.

YNIMG Journal 2014 Journal Article

Non-Negative Spherical Deconvolution (NNSD) for estimation of fiber Orientation Distribution Function in single-/multi-shell diffusion MRI

• Jian Cheng
• Rachid Deriche
• Tianzi Jiang
• Dinggang Shen
• Pew-Thian Yap

Spherical Deconvolution (SD) is commonly used for estimating fiber Orientation Distribution Functions (fODFs) from diffusion-weighted signals. Existing SD methods can be classified into two categories: 1) Continuous Representation based SD (CR-SD), where typically Spherical Harmonic (SH) representation is used for convenient analytical solutions, and 2) Discrete Representation based SD (DR-SD), where the signal profile is represented by a discrete set of basis functions uniformly oriented on the unit sphere. A feasible fODF should be non-negative and should integrate to unity throughout the unit sphere S 2. However, to our knowledge, most existing SH-based SD methods enforce non-negativity only on discretized points and not the whole continuum of S 2. Maximum Entropy SD (MESD) and Cartesian Tensor Fiber Orientation Distributions (CT-FOD) are the only SD methods that ensure non-negativity throughout the unit sphere. They are however computational intensive and are susceptible to errors caused by numerical spherical integration. Existing SD methods are also known to overestimate the number of fiber directions, especially in regions with low anisotropy. DR-SD introduces additional error in peak detection owing to the angular discretization of the unit sphere. This paper proposes a SD framework, called Non-Negative SD (NNSD), to overcome all the limitations above. NNSD is significantly less susceptible to the false-positive peaks, uses SH representation for efficient analytical spherical deconvolution, and allows accurate peak detection throughout the whole unit sphere. We further show that NNSD and most existing SD methods can be extended to work on multi-shell data by introducing a three-dimensional fiber response function. We evaluated NNSD in comparison with Constrained SD (CSD), a quadratic programming variant of CSD, MESD, and an L1-norm regularized non-negative least-squares DR-SD. Experiments on synthetic and real single-/multi-shell data indicate that NNSD improves estimation performance in terms of mean difference of angles, peak detection consistency, and anisotropy contrast between isotropic and anisotropic regions.

YNIMG Journal 2014 Journal Article

Segmentation of neonatal brain MR images using patch-driven level sets

• Li Wang
• Feng Shi
• Gang Li
• Yaozong Gao
• Weili Lin
• John H. Gilmore
• Dinggang Shen

The segmentation of neonatal brain MR image into white matter (WM), gray matter (GM), and cerebrospinal fluid (CSF), is challenging due to the low spatial resolution, severe partial volume effect, high image noise, and dynamic myelination and maturation processes. Atlas-based methods have been widely used for guiding neonatal brain segmentation. Existing brain atlases were generally constructed by equally averaging all the aligned template images from a population. However, such population-based atlases might not be representative of a testing subject in the regions with high inter-subject variability and thus often lead to a low capability in guiding segmentation in those regions. Recently, patch-based sparse representation techniques have been proposed to effectively select the most relevant elements from a large group of candidates, which can be used to generate a subject-specific representation with rich local anatomical details for guiding the segmentation. Accordingly, in this paper, we propose a novel patch-driven level set method for the segmentation of neonatal brain MR images by taking advantage of sparse representation techniques. Specifically, we first build a subject-specific atlas from a library of aligned, manually segmented images by using sparse representation in a patch-based fashion. Then, the spatial consistency in the probability maps from the subject-specific atlas is further enforced by considering the similarities of a patch with its neighboring patches. Finally, the probability maps are integrated into a coupled level set framework for more accurate segmentation. The proposed method has been extensively evaluated on 20 training subjects using leave-one-out cross validation, and also on 132 additional testing subjects. Our method achieved a high accuracy of 0. 919±0. 008 for white matter and 0. 901±0. 005 for gray matter, respectively, measured by Dice ratio for the overlap between the automated and manual segmentations in the cortical region.

YNIMG Journal 2014 Journal Article

Spatial distribution and longitudinal development of deep cortical sulcal landmarks in infants

• Yu Meng
• Gang Li
• Weili Lin
• John H. Gilmore
• Dinggang Shen

Sulcal pits, the locally deepest points in sulci of the highly convoluted and variable cerebral cortex, are found to be spatially consistent across human adult individuals. It is suggested that sulcal pits are genetically controlled and have close relationships with functional areas. To date, the existing imaging studies of sulcal pits are mainly focused on adult brains, yet little is known about the spatial distribution and temporal development of sulcal pits in the first 2years of life, which is the most dynamic and critical period of postnatal brain development. Studying sulcal pits during this period would greatly enrich our limited understandings of the origins and developmental trajectories of sulcal pits, and would also provide important insights into many neurodevelopmental disorders associated with abnormal cortical foldings. In this paper, by using surface-based morphometry, for the first time, we systemically investigated the spatial distribution and temporal development of sulcal pits in major cortical sulci from 73 healthy infants, each with three longitudinal 3T MR scans at term birth, 1year, and 2years of age. Our results suggest that the spatially consistent distributions of sulcal pits in major sulci across individuals have already existed at term birth and this spatial distribution pattern keeps relatively stable in the first 2years of life, despite that the cerebral cortex expands dramatically and the sulcal depth increases considerably during this period. Specially, the depth of sulcal pits increases regionally heterogeneously, with more rapid growth in the high-order association cortex, including the prefrontal and temporal cortices, than the sensorimotor cortex in the first 2years of life. Meanwhile, our results also suggest that there exist hemispheric asymmetries of the spatial distributions of sulcal pits in several cortical regions, such as the central, superior temporal and postcentral sulci, consistently from birth to 2years of age, which likely has close relationships with the lateralization of brain functions of these regions. This study provides detailed insights into the spatial distribution and temporal development of deep sulcal landmarks in infants.

YNIMG Journal 2013 Journal Article

Automatic hippocampus segmentation of 7.0Tesla MR images by combining multiple atlases and auto-context models

• Minjeong Kim
• Guorong Wu
• Wei Li
• Li Wang
• Young-Don Son
• Zang-Hee Cho
• Dinggang Shen

In many neuroscience and clinical studies, accurate measurement of hippocampus is very important to reveal the inter-subject anatomical differences or the subtle intra-subject longitudinal changes due to aging or dementia. Although many automatic segmentation methods have been developed, their performances are still challenged by the poor image contrast of hippocampus in the MR images acquired especially from 1. 5 or 3. 0Tesla (T) scanners. With the recent advance of imaging technology, 7. 0T scanner provides much higher image contrast and resolution for hippocampus study. However, the previous methods developed for segmentation of hippocampus from 1. 5T or 3. 0T images do not work for the 7. 0T images, due to different levels of imaging contrast and texture information. In this paper, we present a learning-based algorithm for automatic segmentation of hippocampi from 7. 0T images, by taking advantages of the state-of-the-art multi-atlas framework and also the auto-context model (ACM). Specifically, ACM is performed in each atlas domain to iteratively construct sequences of location-adaptive classifiers by integrating both image appearance and local context features. Due to the plenty texture information in 7. 0T images, more advanced texture features are also extracted and incorporated into the ACM during the training stage. Then, under the multi-atlas segmentation framework, multiple sequences of ACM-based classifiers are trained for all atlases to incorporate the anatomical variability. In the application stage, for a new image, its hippocampus segmentation can be achieved by fusing the labeling results from all atlases, each of which is obtained by applying the atlas-specific ACM-based classifiers. Experimental results on twenty 7. 0T images with the voxel size of 0. 35×0. 35×0. 35mm3 show very promising hippocampus segmentations (in terms of Dice overlap ratio 89. 1±0. 020), indicating high applicability for the future clinical and neuroscience studies.

YNIMG Journal 2013 Journal Article

Development of cortical anatomical properties from early childhood to early adulthood

• Jingxin Nie
• Gang Li
• Dinggang Shen

Human brain matures in temporal and regional heterogeneity, with some areas matured at early adulthood. In this study, the relationship of cortical structural developments between different cortical sheet regions is systematically analyzed using interregional correlation coefficient and network methods. Specifically, 951 longitudinal T1 brain MR images from 445 healthy subjects with ages from 3 to 20years old are used. The result shows that the development of cortex reaches a turning point at around 7years of age: a) the cortical thickness reaches its highest value and also the cortical folding becomes stable at this age; b) both global and local efficiencies of anatomical correlation networks reach the lowest and highest values at this age, respectively; and c) the change of anatomical correlation networks reach the highest level at this age, and the convergence of different anatomical correlation networks starts to decrease from this age. These results might inspire more studies on why there exists a turning point at this age from different viewpoints. For example, is there any change of synaptic pruning, or is it related to the starting of school life? And how can we benefit from this in the real life?

YNIMG Journal 2013 Journal Article

Multiscale adaptive generalized estimating equations for longitudinal neuroimaging data

• Yimei Li
• John H. Gilmore
• Dinggang Shen
• Martin Styner
• Weili Lin
• Hongtu Zhu

Many large-scale longitudinal imaging studies have been or are being widely conducted to better understand the progress of neuropsychiatric and neurodegenerative disorders and normal brain development. The goal of this article is to develop a multiscale adaptive generalized estimation equation (MAGEE) method for spatial and adaptive analysis of neuroimaging data from longitudinal studies. MAGEE is applicable to making statistical inference on regression coefficients in both balanced and unbalanced longitudinal designs and even in twin and familial studies, whereas standard software platforms have several major limitations in handling these complex studies. Specifically, conventional voxel-based analyses in these software platforms involve Gaussian smoothing imaging data and then independently fitting a statistical model at each voxel. However, the conventional smoothing methods suffer from the lack of spatial adaptivity to the shape and spatial extent of region of interest and the arbitrary choice of smoothing extent, while independently fitting statistical models across voxels does not account for the spatial properties of imaging observations and noise distribution. To address such drawbacks, we adapt a powerful propagation–separation (PS) procedure to sequentially incorporate the neighboring information of each voxel and develop a new novel strategy to solely update a set of parameters of interest, while fixing other nuisance parameters at their initial estimators. Simulation studies and real data analysis show that MAGEE significantly outperforms voxel-based analysis.

YNIMG Journal 2012 Journal Article

A novel framework for longitudinal atlas construction with groupwise registration of subject image sequences

• Shu Liao
• Hongjun Jia
• Guorong Wu
• Dinggang Shen
• The Alzheimer's Disease Neuroimaging Initiative

Longitudinal atlas construction plays an important role in medical image analysis. Given a set of longitudinal images from different subjects, the task of longitudinal atlas construction is to build an atlas sequence which can represent the trend of anatomical changes of the population. The major challenge for longitudinal atlas construction is how to effectively incorporate both the subject-specific information and population information to build the unbiased atlases. In this paper, a novel groupwise longitudinal atlas construction framework is proposed to address this challenge, and the main contributions of the proposed framework lie in the following aspects: (1) The subject-specific longitudinal information is captured by building the growth model for each subject. (2) The longitudinal atlas sequence is constructed by performing groupwise registration among all the subject image sequences, and only one transformation is needed to transform each subject's image sequence to the atlas space. The constructed longitudinal atlases are unbiased and no explicit template is assumed. (3) The proposed method is general, where the number of longitudinal images of each subject and the time points at which they are taken can be different. The proposed method is extensively evaluated on two longitudinal databases, namely the BLSA and ADNI databases, to construct the longitudinal atlas sequence. It is also compared with a state-of-the-art longitudinal atlas construction algorithm based on kernel regression on the temporal domain. Experimental results demonstrate that the proposed method consistently achieves higher registration accuracies and more consistent spatial–temporal correspondences than the compared method on both databases.

YNIMG Journal 2012 Journal Article

Altered structural connectivity in neonates at genetic risk for schizophrenia: A combined study using morphological and white matter networks

• Feng Shi
• Pew-Thian Yap
• Wei Gao
• Weili Lin
• John H. Gilmore
• Dinggang Shen

Recently, an increasing body of evidence suggests that developmental abnormalities related to schizophrenia may occur as early as the neonatal stage. Impairments of brain gray matter and wiring problems of axonal fibers are commonly suspected to be responsible for the disconnection hypothesis in schizophrenia adults, but significantly less is known in neonates. In this study, we investigated 26 neonates who were at genetic risk for schizophrenia and 26 demographically matched healthy neonates using both morphological and white matter networks to examine possible brain connectivity abnormalities. The results showed that both populations exhibited small-world network topology. Morphological network analysis indicated that the brain structural associations of the high-risk neonates tended to have globally lower efficiency, longer connection distance, and less number of hub nodes and edges with relatively higher betweenness. Subgroup analysis showed that male neonates were significantly disease-affected, while the female neonates were not. White matter network analysis, however, showed that the fiber networks were globally unaffected, although several subcortical–cortical connections had significantly less number of fibers in high-risk neonates. This study provides new lines of evidence in support of the disconnection hypothesis, reinforcing the notion that the genetic risk of schizophrenia induces alterations in both gray matter structural associations and white matter connectivity.

YNIMG Journal 2012 Journal Article

Consistent reconstruction of cortical surfaces from longitudinal brain MR images

• Gang Li
• Jingxin Nie
• Guorong Wu
• Yaping Wang
• Dinggang Shen

Accurate and consistent reconstruction of cortical surfaces from longitudinal human brain MR images is of great importance in studying longitudinal subtle change of the cerebral cortex. This paper presents a novel deformable surface method for consistent and accurate reconstruction of inner, central and outer cortical surfaces from longitudinal brain MR images. Specifically, the cortical surfaces of the group-mean image of all aligned longitudinal images of the same subject are first reconstructed by a deformable surface method, which is driven by a force derived from the Laplace's equation. And then the longitudinal cortical surfaces are consistently reconstructed by jointly deforming the cortical surfaces of the group-mean image to all longitudinal images. The proposed method has been successfully applied to two sets of longitudinal human brain MR images. Both qualitative and quantitative experimental results demonstrate the accuracy and consistency of the proposed method. Furthermore, the reconstructed longitudinal cortical surfaces are used to measure the longitudinal changes of cortical thickness in both normal and diseased groups, where the overall decline trend of cortical thickness has been clearly observed. Meanwhile, the longitudinal cortical thickness also shows its potential in distinguishing different clinical groups.

YNIMG Journal 2012 Journal Article

Ensemble sparse classification of Alzheimer's disease

• Manhua Liu
• Daoqiang Zhang
• Dinggang Shen

The high-dimensional pattern classification methods, e. g. , support vector machines (SVM), have been widely investigated for analysis of structural and functional brain images (such as magnetic resonance imaging (MRI)) to assist the diagnosis of Alzheimer's disease (AD) including its prodromal stage, i. e. , mild cognitive impairment (MCI). Most existing classification methods extract features from neuroimaging data and then construct a single classifier to perform classification. However, due to noise and small sample size of neuroimaging data, it is challenging to train only a global classifier that can be robust enough to achieve good classification performance. In this paper, instead of building a single global classifier, we propose a local patch-based subspace ensemble method which builds multiple individual classifiers based on different subsets of local patches and then combines them for more accurate and robust classification. Specifically, to capture the local spatial consistency, each brain image is partitioned into a number of local patches and a subset of patches is randomly selected from the patch pool to build a weak classifier. Here, the sparse representation-based classifier (SRC) method, which has shown to be effective for classification of image data (e. g. , face), is used to construct each weak classifier. Then, multiple weak classifiers are combined to make the final decision. We evaluate our method on 652 subjects (including 198 AD patients, 225 MCI and 229 normal controls) from Alzheimer's Disease Neuroimaging Initiative (ADNI) database using MR images. The experimental results show that our method achieves an accuracy of 90. 8% and an area under the ROC curve (AUC) of 94. 86% for AD classification and an accuracy of 87. 85% and an AUC of 92. 90% for MCI classification, respectively, demonstrating a very promising performance of our method compared with the state-of-the-art methods for AD/MCI classification using MR images.

YNIMG Journal 2012 Journal Article

Erratum to multi-modal multi-task learning for joint prediction of multiple regression and classification variables in Alzheimer's disease [Neuroimage 59/2 (2012) 895–907]

• Daoqiang Zhang
• Dinggang Shen

YNIMG Journal 2012 Journal Article

Identification of MCI individuals using structural and functional connectivity networks

• Chong-Yaw Wee
• Pew-Thian Yap
• Daoqiang Zhang
• Kevin Denny
• Jeffrey N. Browndyke
• Guy G. Potter
• Kathleen A. Welsh-Bohmer
• Lihong Wang

Different imaging modalities provide essential complementary information that can be used to enhance our understanding of brain disorders. This study focuses on integrating multiple imaging modalities to identify individuals at risk for mild cognitive impairment (MCI). MCI, often an early stage of Alzheimer's disease (AD), is difficult to diagnose due to its very mild or insignificant symptoms of cognitive impairment. Recent emergence of brain network analysis has made characterization of neurological disorders at a whole-brain connectivity level possible, thus providing new avenues for brain diseases classification. Employing multiple-kernel Support Vector Machines (SVMs), we attempt to integrate information from diffusion tensor imaging (DTI) and resting-state functional magnetic resonance imaging (rs-fMRI) for improving classification performance. Our results indicate that the multimodality classification approach yields statistically significant improvement in accuracy over using each modality independently. The classification accuracy obtained by the proposed method is 96. 3%, which is an increase of at least 7. 4% from the single modality-based methods and the direct data fusion method. A cross-validation estimation of the generalization performance gives an area of 0. 953 under the receiver operating characteristic (ROC) curve, indicating excellent diagnostic power. The multimodality classification approach hence allows more accurate early detection of brain abnormalities with greater sensitivity.

YNIMG Journal 2012 Journal Article

Iterative multi-atlas-based multi-image segmentation with tree-based registration

• Hongjun Jia
• Pew-Thian Yap
• Dinggang Shen

In this paper, we present a multi-atlas-based framework for accurate, consistent and simultaneous segmentation of a group of target images. Multi-atlas-based segmentation algorithms consider concurrently complementary information from multiple atlases to produce optimal segmentation outcomes. However, the accuracy of these algorithms relies heavily on the precise alignment of the atlases with the target image. In particular, the commonly used pairwise registration may result in inaccurate alignment especially between images with large shape differences. Additionally, when segmenting a group of target images, most current methods consider these images independently with disregard of their correlation, thus resulting in inconsistent segmentations of the same structures across different target images. We propose two novel strategies to address these limitations: 1) a novel tree-based groupwise registration method for concurrent alignment of both the atlases and the target images, and 2) an iterative groupwise segmentation method for simultaneous consideration of segmentation information propagated from all available images, including the atlases and other newly segmented target images. Evaluation based on various datasets indicates that the proposed multi-atlas-based multi-image segmentation (MABMIS) framework yields substantial improvements in terms of consistency and accuracy over methods that do not consider the group of target images holistically.

YNIMG Journal 2012 Journal Article

LABEL: Pediatric brain extraction using learning-based meta-algorithm

• Feng Shi
• Li Wang
• Yakang Dai
• John H. Gilmore
• Weili Lin
• Dinggang Shen

Magnetic resonance imaging of pediatric brain provides valuable information for early brain development studies. Automated brain extraction is challenging due to the small brain size and dynamic change of tissue contrast in the developing brains. In this paper, we propose a novel Learning Algorithm for Brain Extraction and Labeling (LABEL) specially for the pediatric MR brain images. The idea is to perform multiple complementary brain extractions on a given testing image by using a meta-algorithm, including BET and BSE, where the parameters of each run of the meta-algorithm are effectively learned from the training data. Also, the representative subjects are selected as exemplars and used to guide brain extraction of new subjects in different age groups. We further develop a level-set based fusion method to combine multiple brain extractions together with a closed smooth surface for obtaining the final extraction. The proposed method has been extensively evaluated in subjects of three representative age groups, such as neonate (less than 2months), infant (1–2years), and child (5–18years). Experimental results show that, with 45 subjects for training (15 neonates, 15 infant, and 15 children), the proposed method can produce more accurate brain extraction results on 246 testing subjects (75 neonates, 126 infants, and 45 children), i. e. , at average Jaccard Index of 0. 953, compared to those by BET (0. 918), BSE (0. 902), ROBEX (0. 901), GCUT (0. 856), and other fusion methods such as Majority Voting (0. 919) and STAPLE (0. 941). Along with the largely-improved computational efficiency, the proposed method demonstrates its ability of automated brain extraction for pediatric MR images in a large age range.

YNIMG Journal 2012 Journal Article

Multi-modal multi-task learning for joint prediction of multiple regression and classification variables in Alzheimer's disease

• Daoqiang Zhang
• Dinggang Shen

Many machine learning and pattern classification methods have been applied to the diagnosis of Alzheimer's disease (AD) and its prodromal stage, i. e. , mild cognitive impairment (MCI). Recently, rather than predicting categorical variables as in classification, several pattern regression methods have also been used to estimate continuous clinical variables from brain images. However, most existing regression methods focus on estimating multiple clinical variables separately and thus cannot utilize the intrinsic useful correlation information among different clinical variables. On the other hand, in those regression methods, only a single modality of data (usually only the structural MRI) is often used, without considering the complementary information that can be provided by different modalities. In this paper, we propose a general methodology, namely multi-modal multi-task (M3T) learning, to jointly predict multiple variables from multi-modal data. Here, the variables include not only the clinical variables used for regression but also the categorical variable used for classification, with different tasks corresponding to prediction of different variables. Specifically, our method contains two key components, i. e. , (1) a multi-task feature selection which selects the common subset of relevant features for multiple variables from each modality, and (2) a multi-modal support vector machine which fuses the above-selected features from all modalities to predict multiple (regression and classification) variables. To validate our method, we perform two sets of experiments on ADNI baseline MRI, FDG-PET, and cerebrospinal fluid (CSF) data from 45 AD patients, 91 MCI patients, and 50 healthy controls (HC). In the first set of experiments, we estimate two clinical variables such as Mini Mental State Examination (MMSE) and Alzheimer's Disease Assessment Scale-Cognitive Subscale (ADAS-Cog), as well as one categorical variable (with value of ‘AD’, ‘MCI’ or ‘HC’), from the baseline MRI, FDG-PET, and CSF data. In the second set of experiments, we predict the 2-year changes of MMSE and ADAS-Cog scores and also the conversion of MCI to AD from the baseline MRI, FDG-PET, and CSF data. The results on both sets of experiments demonstrate that our proposed M3T learning scheme can achieve better performance on both regression and classification tasks than the conventional learning methods.

YNIMG Journal 2012 Journal Article

Registration of longitudinal brain image sequences with implicit template and spatial–temporal heuristics

• Guorong Wu
• Qian Wang
• Dinggang Shen

Accurate measurement of longitudinal changes of brain structures and functions is very important but challenging in many clinical studies. Also, across-subject comparison of longitudinal changes is critical in identifying disease-related changes. In this paper, we propose a novel method to meet these two requirements by simultaneously registering sets of longitudinal image sequences of different subjects to the common space, without assuming any explicit template. Specifically, our goal is to 1) consistently measure the longitudinal changes from a longitudinal image sequence of each subject, and 2) jointly align all image sequences of different subjects to a hidden common space. To achieve these two goals, we first introduce a set of temporal fiber bundles to explore the spatial–temporal behavior of anatomical changes in each longitudinal image sequence. Then, a probabilistic model is built upon the temporal fibers to characterize both spatial smoothness and temporal continuity. Finally, the transformation fields that connect each time-point image of each subject to the common space are simultaneously estimated by the expectation maximization (EM) approach, via the maximum a posterior (MAP) estimation of the probabilistic models. Promising results have been obtained in quantitative measurement of longitudinal brain changes, i. e. , hippocampus volume changes, showing better performance than those obtained by either the pairwise or the groupwise only registration methods.

YNIMG Journal 2011 Journal Article

Automatic segmentation of neonatal images using convex optimization and coupled level sets

• Li Wang
• Feng Shi
• Weili Lin
• John H. Gilmore
• Dinggang Shen

Accurate segmentation of neonatal brain MR images remains challenging mainly due to their poor spatial resolution, inverted contrast between white matter and gray matter, and high intensity inhomogeneity. Most existing methods for neonatal brain segmentation are atlas-based and voxel-wise. Although active contour/surface models with geometric information constraint have been successfully applied to adult brain segmentation, they are not fully explored in the neonatal image segmentation. In this paper, we propose a novel neonatal image segmentation method by combining local intensity information, atlas spatial prior, and cortical thickness constraint in a single level-set framework. Besides, we also provide a robust and reliable tissue surface initialization for the proposed method by using a convex optimization technique. Thus, tissue segmentation, as well as inner and outer cortical surface reconstruction, can be obtained simultaneously. The proposed method has been tested on a large neonatal dataset, and the validation on 10 neonatal brain images (with manual segmentations) shows very promising results.

YNIMG Journal 2011 Journal Article

Brain anatomical networks in early human brain development

• Yong Fan
• Feng Shi
• Jeffrey Keith Smith
• Weili Lin
• John H. Gilmore
• Dinggang Shen

Recent neuroimaging studies have demonstrated that human brain networks have economic small-world topology and modular organization, enabling efficient information transfer among brain regions. However, it remains largely unknown how the small-world topology and modular organization of human brain networks emerge and develop. Using longitudinal MRI data of 28 healthy pediatric subjects, collected at their ages of 1month, 1year, and 2years, we analyzed development patterns of brain anatomical networks derived from morphological correlations of brain regional volumes. The results show that the brain network of 1-month-olds has the characteristically economic small-world topology and nonrandom modular organization. The network's cost efficiency increases with the brain development to 1year and 2years, so does the modularity, providing supportive evidence for the hypothesis that the small-world topology and the modular organization of brain networks are established during early brain development to support rapid synchronization and information transfer with minimal rewiring cost, as well as to balance between local processing and global integration of information.

YNIMG Journal 2011 Journal Article

Consistent sulcal parcellation of longitudinal cortical surfaces

• Gang Li
• Dinggang Shen

Automated accurate and consistent sulcal parcellation of longitudinal cortical surfaces is of great importance in studying longitudinal morphological and functional changes of human brains, since longitudinal cortical changes are normally very subtle, especially in aging brains. However, applying the existing methods (which were typically developed for cortical sulcal parcellation of a single cortical surface) independently to longitudinal cortical surfaces might generate longitudinally-inconsistent results. To overcome this limitation, this paper presents a novel energy function based method for accurate and consistent sulcal parcellation of longitudinal cortical surfaces. Specifically, both spatial and temporal smoothness are imposed in the energy function to obtain consistent longitudinal sulcal parcellation results. The energy function is efficiently minimized by a graph cut method. The proposed method has been successfully applied to sulcal parcellation of both real and simulated longitudinal inner cortical surfaces of human brain MR images. Both qualitative and quantitative evaluation results demonstrate the validity of the proposed method.

YNIMG Journal 2011 Journal Article

Enriched white matter connectivity networks for accurate identification of MCI patients

• Chong-Yaw Wee
• Pew-Thian Yap
• Wenbin Li
• Kevin Denny
• Jeffrey N. Browndyke
• Guy G. Potter
• Kathleen A. Welsh-Bohmer
• Lihong Wang

Mild cognitive impairment (MCI), often a prodromal phase of Alzheimer's disease (AD), is frequently considered to be a good target for early diagnosis and therapeutic interventions of AD. Recent emergence of reliable network characterization techniques has made it possible to understand neurological disorders at a whole-brain connectivity level. Accordingly, we propose an effective network-based multivariate classification algorithm, using a collection of measures derived from white matter (WM) connectivity networks, to accurately identify MCI patients from normal controls. An enriched description of WM connections, utilizing six physiological parameters, i. e. , fiber count, fractional anisotropy (FA), mean diffusivity (MD), and principal diffusivities(λ 1, λ 2, and λ 3), results in six connectivity networks for each subject to account for the connection topology and the biophysical properties of the connections. Upon parcellating the brain into 90 regions-of-interest (ROIs), these properties can be quantified for each pair of regions with common traversing fibers. For building an MCI classifier, clustering coefficient of each ROI in relation to the remaining ROIs is extracted as feature for classification. These features are then ranked according to their Pearson correlation with respect to the clinical labels, and are further sieved to select the most discriminant subset of features using an SVM-based feature selection algorithm. Finally, support vector machines (SVMs) are trained using the selected subset of features. Classification accuracy was evaluated via leave-one-out cross-validation to ensure generalization of performance. The classification accuracy given by our enriched description of WM connections is 88. 9%, which is an increase of at least 14. 8% from that using simple WM connectivity description with any single physiological parameter. A cross-validation estimation of the generalization performance shows an area of 0. 929 under the receiver operating characteristic (ROC) curve, indicating excellent diagnostic power. It was also found, based on the selected features, that portions of the prefrontal cortex, orbitofrontal cortex, parietal lobe and insula regions provided the most discriminant features for classification, in line with results reported in previous studies. Our MCI classification framework, especially the enriched description of WM connections, allows accurate early detection of brain abnormalities, which is of paramount importance for treatment management of potential AD patients.

YNIMG Journal 2011 Journal Article

Intermediate templates guided groupwise registration of diffusion tensor images

• Hongjun Jia
• Pew-Thian Yap
• Guorong Wu
• Qian Wang
• Dinggang Shen

Registration of a population of diffusion tensor images (DTIs) is one of the key steps in medical image analysis, and it plays an important role in the statistical analysis of white matter related neurological diseases. However, pairwise registration with respect to a pre-selected template may not give precise results if the selected template deviates significantly from the distribution of images. To cater for more accurate and consistent registration, a novel framework is proposed for groupwise registration with the guidance from one or more intermediate templates determined from the population of images. Specifically, we first use a Euclidean distance, defined as a combinative measure based on the FA map and ADC map, for gauging the similarity of each pair of DTIs. A fully connected graph is then built with each node denoting an image and each edge denoting the distance between a pair of images. The root template image is determined automatically as the image with the overall shortest path length to all other images on the minimum spanning tree (MST) of the graph. Finally, a sequence of registration steps is applied to progressively warping each image towards the root template image with the help of intermediate templates distributed along its path to the root node on the MST. Extensive experimental results using diffusion tensor images of real subjects indicate that registration accuracy and fiber tract alignment are significantly improved, compared with the direct registration from each image to the root template image.

YNIMG Journal 2011 Journal Article

Longitudinal regression analysis of spatial–temporal growth patterns of geometrical diffusion measures in early postnatal brain development with diffusion tensor imaging

• Yasheng Chen
• Hongyu An
• Hongtu Zhu
• Valerie Jewells
• Diane Armao
• Dinggang Shen
• John H. Gilmore
• Weili Lin

Although diffusion tensor imaging (DTI) has provided substantial insights into early brain development, most DTI studies based on fractional anisotropy (FA) and mean diffusivity (MD) may not capitalize on the information derived from the three principal diffusivities (e. g. eigenvalues). In this study, we explored the spatial and temporal evolution of white matter structures during early brain development using two geometrical diffusion measures, namely, linear (Cl) and planar (Cp) diffusion anisotropies, from 71 longitudinal datasets acquired from 29 healthy, full-term pediatric subjects. The growth trajectories were estimated with generalized estimating equations (GEE) using linear fitting with logarithm of age (days). The presence of the white matter structures in Cl and Cp was observed in neonates, suggesting that both the cylindrical and fanning or crossing structures in various white matter regions may already have been formed at birth. Moreover, we found that both Cl and Cp evolved in a temporally nonlinear and spatially inhomogeneous manner. The growth velocities of Cl in central white matter were significantly higher when compared to peripheral, or more laterally located, white matter: central growth velocity Cl =0. 0465±0. 0273/log(days), versus peripheral growth velocity Cl =0. 0198±0. 0127/log(days), p<10−6. In contrast, the growth velocities of Cp in central white matter were significantly lower than that in peripheral white matter: central growth velocity Cp= 0. 0014 ± 0. 0058/log(days), versus peripheral growth velocity Cp =0. 0289±0. 0101/log(days), p<10−6. Depending on the underlying white matter site which is analyzed, our findings suggest that ongoing physiologic and microstructural changes in the developing brain may exert different effects on the temporal evolution of these two geometrical diffusion measures. Thus, future studies utilizing DTI with correlative histological analysis in the study of early brain development are warranted.

YNIMG Journal 2011 Journal Article

Multimodal classification of Alzheimer's disease and mild cognitive impairment

• Daoqiang Zhang
• Yaping Wang
• Luping Zhou
• Hong Yuan
• Dinggang Shen

Effective and accurate diagnosis of Alzheimer's disease (AD), as well as its prodromal stage (i. e. , mild cognitive impairment (MCI)), has attracted more and more attention recently. So far, multiple biomarkers have been shown to be sensitive to the diagnosis of AD and MCI, i. e. , structural MR imaging (MRI) for brain atrophy measurement, functional imaging (e. g. , FDG-PET) for hypometabolism quantification, and cerebrospinal fluid (CSF) for quantification of specific proteins. However, most existing research focuses on only a single modality of biomarkers for diagnosis of AD and MCI, although recent studies have shown that different biomarkers may provide complementary information for the diagnosis of AD and MCI. In this paper, we propose to combine three modalities of biomarkers, i. e. , MRI, FDG-PET, and CSF biomarkers, to discriminate between AD (or MCI) and healthy controls, using a kernel combination method. Specifically, ADNI baseline MRI, FDG-PET, and CSF data from 51AD patients, 99 MCI patients (including 43 MCI converters who had converted to AD within 18months and 56 MCI non-converters who had not converted to AD within 18months), and 52 healthy controls are used for development and validation of our proposed multimodal classification method. In particular, for each MR or FDG-PET image, 93 volumetric features are extracted from the 93 regions of interest (ROIs), automatically labeled by an atlas warping algorithm. For CSF biomarkers, their original values are directly used as features. Then, a linear support vector machine (SVM) is adopted to evaluate the classification accuracy, using a 10-fold cross-validation. As a result, for classifying AD from healthy controls, we achieve a classification accuracy of 93. 2% (with a sensitivity of 93% and a specificity of 93. 3%) when combining all three modalities of biomarkers, and only 86. 5% when using even the best individual modality of biomarkers. Similarly, for classifying MCI from healthy controls, we achieve a classification accuracy of 76. 4% (with a sensitivity of 81. 8% and a specificity of 66%) for our combined method, and only 72% even using the best individual modality of biomarkers. Further analysis on MCI sensitivity of our combined method indicates that 91. 5% of MCI converters and 73. 4% of MCI non-converters are correctly classified. Moreover, we also evaluate the classification performance when employing a feature selection method to select the most discriminative MR and FDG-PET features. Again, our combined method shows considerably better performance, compared to the case of using an individual modality of biomarkers.

YNIMG Journal 2011 Journal Article

Sex differences in grey matter atrophy patterns among AD and aMCI patients: Results from ADNI

• Martha Skup
• Hongtu Zhu
• Yaping Wang
• Kelly S. Giovanello
• Ja-an Lin
• Dinggang Shen
• Feng Shi
• Wei Gao

We used longitudinal magnetic resonance imaging (MRI) data to determine whether there are any gender differences in grey matter atrophy patterns over time in 197 individuals with probable Alzheimer's disease (AD) and 266 with amnestic mild cognitive impairment (aMCI), compared with 224 healthy controls participating in the Alzheimer's Disease Neuroimaging Initiative (ADNI). While previous research has differentiated probable AD and aMCI groups from controls in brain atrophy, it is unclear whether and how sex plays a role in patterns of change over time. Using regional volumetric maps, we fit longitudinal models to the grey matter data collected at repeated occasions, seeking differences in patterns of volume change over time by sex and diagnostic group in a voxel-wise analysis. Additionally, using a region-of-interest approach, we fit longitudinal models to the global volumetric data of predetermined brain regions to determine whether this more conventional approach is sufficient for determining sex and group differences in atrophy. Our longitudinal analyses revealed that, of the various grey matter regions investigated, males and females in the AD group and the aMCI group showed different patterns of decline over time compared to controls in the bilateral precuneus, bilateral caudate nucleus, right entorhinal gyrus, bilateral thalamus, bilateral middle temporal gyrus, left insula, and right amygdala. As one of the first investigation to model more than two time points of structural MRI data over time, our findings add insight into how AD and aMCI males and females differ from controls and from each other over time.

YNIMG Journal 2011 Journal Article

SharpMean: Groupwise registration guided by sharp mean image and tree-based registration

• Guorong Wu
• Hongjun Jia
• Qian Wang
• Dinggang Shen

Groupwise registration has become more and more popular due to its attractiveness for unbiased analysis of population data. One of the most popular approaches for groupwise registration is to iteratively calculate the group mean image and then register all subject images towards the latest estimated group mean image. However, its performance might be undermined by the fuzzy mean image estimated in the very beginning of groupwise registration procedure, because all subject images are far from being well-aligned at that moment. In this paper, we first point out the significance of always keeping the group mean image sharp and clear throughout the entire groupwise registration procedure, which is intuitively important but has not been explored in the literature yet. To achieve this, we resort to developing the robust mean-image estimator by the adaptive weighting strategy, where the weights are adaptive across not only the individual subject images but also all spatial locations in the image domain. On the other hand, we notice that some subjects might have large anatomical variations from the group mean image, which challenges most of the state-of-the-art registration algorithms. To ensure good registration results in each iteration, we explore the manifold of subject images and build a minimal spanning tree (MST) with the group mean image as the root of the MST. Therefore, each subject image is only registered to its parent node often with similar shapes, and its overall transformation to the group mean image space is obtained by concatenating all deformations along the paths connecting itself to the root of the MST (the group mean image). As a result, all the subjects will be well aligned to the group mean image adaptively. Our method has been evaluated in both real and simulated datasets. In all experiments, our method outperforms the conventional algorithm which generally produces a fuzzy group mean image throughout the entire groupwise registration.

YNIMG Journal 2011 Journal Article

SPHERE: SPherical Harmonic Elastic REgistration of HARDI data

• Pew-Thian Yap
• Yasheng Chen
• Hongyu An
• Yang Yang
• John H. Gilmore
• Weili Lin
• Dinggang Shen

In contrast to the more common Diffusion Tensor Imaging (DTI), High Angular Resolution Diffusion Imaging (HARDI) allows superior delineation of angular microstructures of brain white matter, and makes possible multiple-fiber modeling of each voxel for better characterization of brain connectivity. However, the complex orientation information afforded by HARDI makes registration of HARDI images more complicated than scalar images. In particular, the question of how much orientation information is needed for satisfactory alignment has not been sufficiently addressed. Low order orientation representation is generally more robust than high order representation, although the latter provides more information for correct alignment of fiber pathways. However, high order representation, when naïvely utilized, might not necessarily be conducive to improving registration accuracy since similar structures with significant orientation differences prior to proper alignment might be mistakenly taken as non-matching structures. We present in this paper a HARDI registration algorithm, called SPherical Harmonic Elastic REgistration (SPHERE), which in a principled means hierarchically extracts orientation information from HARDI data for structural alignment. The image volumes are first registered using robust, relatively direction invariant features derived from the Orientation Distribution Function (ODF), and the alignment is then further refined using spherical harmonic (SH) representation with gradually increasing orders. This progression from non-directional, single-directional to multi-directional representation provides a systematic means of extracting directional information given by diffusion-weighted imaging. Coupled with a template–subject-consistent soft-correspondence-matching scheme, this approach allows robust and accurate alignment of HARDI data. Experimental results show marked increase in accuracy over a state-of-the-art DTI registration algorithm.

YNIMG Journal 2010 Journal Article

ABSORB: Atlas building by self-organized registration and bundling

• Hongjun Jia
• Guorong Wu
• Qian Wang
• Dinggang Shen

To achieve more accurate and consistent registration in an image population, a novel hierarchical groupwise registration framework, called Atlas Building by Self-Organized Registration and Bundling (ABSORB), is proposed in this paper. In this new framework, the global structure, i. e. , the relative distribution of subject images is always preserved during the registration process by constraining each subject image to deform only locally with respect to its neighbors within the learned image manifold. To achieve this goal, two novel strategies, i. e. , the self-organized registration by warping one image towards a set of its eligible neighbors and image bundling to cluster similar images, are specially proposed. By using these two strategies, this new framework can perform groupwise registration in a hierarchical way. Specifically, in the high level, it will perform on a much smaller dataset formed by the representative subject images of all subgroups that are generated in the previous levels of registration. Compared to the other groupwise registration methods, our proposed framework has several advantages: (1) it explores the local data distribution and uses the obtained distribution information to guide the registration; (2) the possible registration error can be greatly reduced by requiring each individual subject to move only towards its nearby subjects with similar structures; (3) it can produce a smoother registration path, in general, from each subject image to the final built atlas than other groupwise registration methods. Experimental results on both synthetic and real datasets show that the proposed framework can achieve substantial improvements, compared to the other two widely used groupwise registration methods, in terms of both registration accuracy and robustness.

YNIMG Journal 2010 Journal Article

Attribute vector guided groupwise registration

• Qian Wang
• Guorong Wu
• Pew-Thian Yap
• Dinggang Shen

Groupwise registration has been recently introduced to simultaneously register a group of images by avoiding the selection of a particular template. To achieve this, several methods have been proposed to take advantage of information-theoretic entropy measures based on image intensity. However, simplistic utilization of voxelwise image intensity is not sufficient to establish reliable correspondences, since it lacks important contextual information. Therefore, we explore the notion of attribute vector as the voxel signature, instead of image intensity, to guide the correspondence detection in groupwise registration. In particular, for each voxel, the attribute vector is computed from its multi-scale neighborhoods, in order to capture the geometric information at different scales. The probability density function (PDF) of each element in the attribute vector is then estimated from the local neighborhood, providing a statistical summary of the underlying anatomical structure in that local pattern. Eventually, with the help of Jensen–Shannon (JS) divergence, a group of subjects can be aligned simultaneously by minimizing the sum of JS divergences across the image domain and all attributes. We have employed our groupwise registration algorithm on both real (NIREP NA0 data set) and simulated data (12 pairs of normal control and simulated atrophic data set). The experimental results demonstrate that our method yields better registration accuracy, compared with a popular groupwise registration method.

YNIMG Journal 2010 Journal Article

Construction of multi-region-multi-reference atlases for neonatal brain MRI segmentation

• Feng Shi
• Pew-Thian Yap
• Yong Fan
• John H. Gilmore
• Weili Lin
• Dinggang Shen

Neonatal brain MRI segmentation is a challenging problem due to its poor image quality. Atlas-based segmentation approaches have been widely used for guiding brain tissue segmentation. Existing brain atlases are usually constructed by equally averaging pre-segmented images in a population. However, such approaches diminish local inter-subject structural variability and thus lead to lower segmentation guidance capability. To deal with this problem, we propose a multi-region-multi-reference framework for atlas-based neonatal brain segmentation. For each region of a brain parcellation, a population of spatially normalized pre-segmented images is clustered into a number of sub-populations. Each sub-population of a region represents an independent distribution from which a regional probability atlas can be generated. A selection of these regional atlases, across different sub-regions, will in the end be adaptively combined to form an overall atlas specific to the query image. Given a query image, the determination of the appropriate set of regional atlases is achieved by comparing the query image regionally with the reference, or exemplar, of each sub-population. Upon obtaining an overall atlas, an atlas-based joint registration–segmentation strategy is employed to segment the query image. Since the proposed method generates an atlas which is significant more similar to the query image than the traditional average-shape atlas, better tissue segmentation results can be expected. This is validated by applying the proposed method on a large set of neonatal brain images available in our institute. Experimental results on a randomly selected set of 10 neonatal brain images indicate that the proposed method achieves higher tissue overlap rates and lower standard deviations (SDs) in comparison with manual segmentations, i. e. , 0. 86 (SD 0. 02) for GM, 0. 83 (SD 0. 03) for WM, and 0. 80 (SD 0. 05) for CSF. The proposed method also outperforms two other average-shape atlas-based segmentation methods.

YNIMG Journal 2010 Journal Article

Neonatal brain image segmentation in longitudinal MRI studies

• Feng Shi
• Yong Fan
• Songyuan Tang
• John H. Gilmore
• Weili Lin
• Dinggang Shen

In the study of early brain development, tissue segmentation of neonatal brain MR images remains challenging because of the insufficient image quality due to the properties of developing tissues. Among various brain tissue segmentation algorithms, atlas-based brain image segmentation can potentially achieve good segmentation results on neonatal brain images. However, their performances rely on both the quality of the atlas and the spatial correspondence between the atlas and the to-be-segmented image. Moreover, it is difficult to build a population atlas for neonates due to the requirement of a large set of tissue-segmented neonatal brain images. To combat these obstacles, we present a longitudinal neonatal brain image segmentation framework by taking advantage of the longitudinal data acquired at late time-point to build a subject-specific tissue probabilistic atlas. Specifically, tissue segmentation of the neonatal brain is formulated as two iterative steps of bias correction and probabilistic-atlas-based tissue segmentation, along with the longitudinal atlas reconstructed by the late time image of the same subject. The proposed method has been evaluated qualitatively through visual inspection and quantitatively by comparing with manual delineations and two population-atlas-based segmentation methods. Experimental results show that the utilization of a subject-specific probabilistic atlas can substantially improve tissue segmentation of neonatal brain images.

YNIMG Journal 2010 Journal Article

TPS-HAMMER: Improving HAMMER registration algorithm by soft correspondence matching and thin-plate splines based deformation interpolation

• Guorong Wu
• Pew-Thian Yap
• Minjeong Kim
• Dinggang Shen

We present an improved MR brain image registration algorithm, called TPS-HAMMER, which is based on the concepts of attribute vectors and hierarchical landmark selection scheme proposed in the highly successful HAMMER registration algorithm. We demonstrate that TPS-HAMMER algorithm yields better registration accuracy, robustness, and speed over HAMMER owing to (1) the employment of soft correspondence matching and (2) the utilization of thin-plate splines (TPS) for sparse-to-dense deformation field generation. These two aspects can be integrated into a unified framework to refine the registration iteratively by alternating between soft correspondence matching and dense deformation field estimation. Compared with HAMMER, TPS-HAMMER affords several advantages: (1) unlike the Gaussian propagation mechanism employed in HAMMER, which can be slow and often leaves unreached blotches in the deformation field, the deformation interpolation in the non-landmark points can be obtained immediately with TPS in our algorithm; (2) the smoothness of deformation field is preserved due to the nice properties of TPS; (3) possible misalignments can be alleviated by allowing the matching of the landmarks with a number of possible candidate points and enforcing more exact matches in the final stages of the registration. Extensive experiments have been conducted, using the original HAMMER as a comparison baseline, to validate the merits of TPS-HAMMER. The results show that TPS-HAMMER yields significant improvement in both accuracy and speed, indicating high applicability for the clinical scenario.

YNIMG Journal 2009 Journal Article

Corrigendum to “Non-diffeomorphic registration of brain tumor images by simulating tissue loss and tumor growth” [NeuroImage 46 (2009) 762–774]

• Evangelia I. Zacharaki
• Cosmina S. Hogea
• Dinggang Shen
• George Biros
• Christos Davatzikos

YNIMG Journal 2009 Journal Article

Non-diffeomorphic registration of brain tumor images by simulating tissue loss and tumor growth

• Evangelia I. Zacharaki
• Cosmina S. Hogea
• Dinggang Shen
• George Biros
• Christos Davatzikos

Although a variety of diffeomorphic deformable registration methods exist in the literature, application of these methods in the presence of space-occupying lesions is not straightforward. The motivation of this work is spatial normalization of MR images from patients with brain tumors in a common stereotaxic space, aiming to pool data from different patients into a common space in order to perform group analyses. Additionally, transfer of structural and functional information from neuroanatomical brain atlases into the individual patient's space can be achieved via the inverse mapping, for the purpose of segmenting brains and facilitating surgical or radiotherapy treatment planning. A method that estimates the brain tissue loss and replacement by tumor is applied for achieving equivalent image content between an atlas and a patient's scan, based on a biomechanical model of tumor growth. Automated estimation of the parameters modeling brain tissue loss and displacement is performed via optimization of an objective function reflecting feature-based similarity and elastic stretching energy, which is optimized in parallel via APPSPACK (Asynchronous Parallel Pattern Search). The results of the method, applied to 21 brain tumor patients, indicate that the registration accuracy is relatively high in areas around the tumor, as well as in the healthy portion of the brain. Also, the calculated deformation in the vicinity of the tumor is shown to correlate highly with expert-defined visual scores indicating the tumor mass effect, thereby potentially leading to an objective approach to quantification of mass effect, which is commonly used in diagnosis.

YNIMG Journal 2009 Journal Article

RABBIT: Rapid alignment of brains by building intermediate templates

• Songyuan Tang
• Yong Fan
• Guorong Wu
• Minjeong Kim
• Dinggang Shen

A brain image registration algorithm, referred to as RABBIT, is proposed to achieve fast and accurate image registration with the help of an intermediate template generated by a statistical deformation model. The statistical deformation model is built by principal component analysis (PCA) on a set of training samples of brain deformation fields that warp a selected template image to the individual brain samples. The statistical deformation model is capable of characterizing individual brain deformations by a small number of parameters, which is used to rapidly estimate the brain deformation between the template and a new individual brain image. The estimated deformation is then used to warp the template, thus generating an intermediate template close to the individual brain image. Finally, the shape difference between the intermediate template and the individual brain is estimated by an image registration algorithm, e. g. , HAMMER. The overall registration between the template and the individual brain image can be achieved by directly combining the deformation fields that warp the template to the intermediate template, and the intermediate template to the individual brain image. The algorithm has been validated for spatial normalization of both simulated and real magnetic resonance imaging (MRI) brain images. Compared with HAMMER, the experimental results demonstrate that the proposed algorithm can achieve over five times speedup, with similar registration accuracy and statistical power in detecting brain atrophy.

YNIMG Journal 2009 Journal Article

TIMER: Tensor Image Morphing for Elastic Registration

• Pew-Thian Yap
• Guorong Wu
• Hongtu Zhu
• Weili Lin
• Dinggang Shen

We propose a novel diffusion tensor imaging (DTI) registration algorithm, called Tensor Image Morphing for Elastic Registration (TIMER), which leverages the hierarchical guidance of regional distributions and local boundaries, both extracted directly from the tensors. Currently available DTI registration methods generally extract tensor scalar features from each tensor to construct scalar maps. Subsequently, regional integration and other operations such as edge detection are performed to extract more features to guide the registration. However, there are two major limitations with these approaches. First, the computed regional features might not reflect the actual regional tensor distributions. Second, by the same token, gradient maps calculated from the tensor-derived scalar feature maps might not represent the actual tissue tensor boundaries. To overcome these limitations, we propose a new approach which extracts regional and edge information directly from a tensor neighborhood. Regional tensor distribution information, such as mean and variance, is computed in a multiscale fashion directly from the tensors by taking into account the voxel neighborhood of different sizes, and hence capturing tensor information at different scales, which in turn can be employed to hierarchically guide the registration. Such multiscale scheme can help alleviate the problem of local minimum and is also more robust to noise since one can better determine the statistical properties of each voxel by taking into account the properties of its surrounding. Also incorporated in our method is edge information extracted directly from the tensors, which is crucial to facilitate registration of tissue boundaries. Experiments involving real subjects, simulated subjects, fiber tracking, and atrophy detection indicate that TIMER performs better than the other methods (Yang et al. , 2008; Zhang et al. , 2006).

YNIMG Journal 2009 Journal Article

White matter abnormalities revealed by diffusion tensor imaging in non-demented and demented HIV+ patients

• Yasheng Chen
• Hongyu An
• Hongtu Zhu
• Taylor Stone
• J. Keith Smith
• Colin Hall
• Elizabeth Bullitt
• Dinggang Shen

HIV associated dementia (HAD) is the most advanced stage of central nervous system disease caused by HIV infection. Previous studies have demonstrated that patients with HAD exhibit greater cerebral and basal ganglia atrophy than non-demented HIV+ (HND) patients. However, the extent to which white matter is affected in HAD patients compared to HND patients remains elusive. This study is designed to address the potential white matter abnormalities through the utilization of diffusion tensor imaging (DTI) in both HND and HAD patients. DTI and T1-weighted images were acquired from 18 healthy controls, 21 HND and 8 HAD patients. T1 image-based registration was performed to 1) parcellate the whole brain white matter into major white matter regions, including frontal, parietal, temporal and occipital white matter, corpus callosum and internal capsule for statistical comparisons of the mean DTI values, and 2) warp all DTI parametric images towards the common template space for voxel-based analysis. The statistical comparisons were performed with four DTI parameters including fractional anisotropy (FA), mean (MD), axial (AD), and radial (RD) diffusivities. With Whitney U tests on the mean DTI values, both HND and HAD demonstrated significant differences from the healthy control in multiple white matter regions. In addition, HAD patients exhibited significantly elevated MD and RD in the parietal white matter when compared to HND patients. In the voxel-based analysis, widespread abnormal regions were identified for both HND and HAD patients, although a much larger abnormal volume was observed in HAD patients for all four DTI parameters. Furthermore, both region of interest (ROI) based and voxel-based analyses revealed that RD was affected to a much greater extent than AD by HIV infection, which may suggest that demyelination is the prominent disease progression in white matter.

YNIMG Journal 2007 Journal Article

Multivariate examination of brain abnormality using both structural and functional MRI

• Yong Fan
• Hengyi Rao
• Hallam Hurt
• Joan Giannetta
• Marc Korczykowski
• David Shera
• Brian B. Avants
• James C. Gee

A multivariate classification approach has been presented to examine the brain abnormalities, i. e. , due to prenatal cocaine exposure, using both structural and functional brain images. First, a regional statistical feature extraction scheme was adopted to capture discriminative features from voxel-wise morphometric and functional representations of brain images, in order to reduce the dimensionality of the features used for classification, as well as to achieve the robustness to registration error and inter-subject variations. Then, this feature extraction method was used in conjunction with a hybrid feature selection method and a nonlinear support vector machine for the classification of brain abnormalities. This brain classification approach has been applied to detecting the brain abnormality associated with prenatal cocaine exposure in adolescents. A promising classification performance was achieved on a data set of 49 subjects (24 normal and 25 prenatally cocaine-exposed teenagers), with a leave-one-out cross-validation. Experimental results demonstrated the efficacy of our method, as well as the importance of incorporating both structural and functional images for brain classification. Moreover, spatial patterns of group difference derived from the constructed classifier were mostly consistent with the results of the conventional statistical analysis method. Therefore, the proposed approach provided not only a multivariate classification method for detecting brain abnormalities, but also an alternative way for group analysis of multimodality images.

YNIMG Journal 2006 Journal Article

CLASSIC: Consistent Longitudinal Alignment and Segmentation for Serial Image Computing

• Zhong Xue
• Dinggang Shen
• Christos Davatzikos

This paper proposes a temporally consistent and spatially adaptive longitudinal MR brain image segmentation algorithm, referred to as CLASSIC, which aims at obtaining accurate measurements of rates of change of regional and global brain volumes from serial MR images. The algorithm incorporates image-adaptive clustering, spatiotemporal smoothness constraints, and image warping to jointly segment a series of 3-D MR brain images of the same subject that might be undergoing changes due to development, aging, or disease. Morphological changes, such as growth or atrophy, are also estimated as part of the algorithm. Experimental results on simulated and real longitudinal MR brain images show both segmentation accuracy and longitudinal consistency.

YNIMG Journal 2006 Journal Article

Simulating deformations of MR brain images for validation of atlas-based segmentation and registration algorithms

• Zhong Xue
• Dinggang Shen
• Bilge Karacali
• Joshua Stern
• David Rottenberg
• Christos Davatzikos

Simulated deformations and images can act as the gold standard for evaluating various template-based image segmentation and registration algorithms. Traditional deformable simulation methods, such as the use of analytic deformation fields or the displacement of landmarks followed by some form of interpolation, are often unable to construct rich (complex) and/or realistic deformations of anatomical organs. This paper presents new methods aiming to automatically simulate realistic inter- and intra-individual deformations. The paper first describes a statistical approach to capturing inter-individual variability of high-deformation fields from a number of examples (training samples). In this approach, Wavelet–Packet Transform (WPT) of the training deformations and their Jacobians, in conjunction with a Markov random field (MRF) spatial regularization, are used to capture both coarse and fine characteristics of the training deformations in a statistical fashion. Simulated deformations can then be constructed by randomly sampling the resultant statistical distribution in an unconstrained or a landmark-constrained fashion. The paper also describes a model for generating tissue atrophy or growth in order to simulate intra-individual brain deformations. Several sets of simulated deformation fields and respective images are generated, which can be used in the future for systematic and extensive validation studies of automated atlas-based segmentation and deformable registration methods. The code and simulated data are available through our Web site.

YNIMG Journal 2004 Journal Article

Automated morphometric study of brain variation in XXY males

• Dinggang Shen
• Dengfeng Liu
• Hong Liu
• Liv Clasen
• Jay Giedd
• Christos Davatzikos

This paper studies brain morphometry variation associated with XXY males (Klinefelter's syndrome) by using an automated whole-brain volumetric analysis method. The application to 34 XXY males and 62 normal male controls reveals pronounced volume reduction in the brains of XXY males, relative to the brains of normal controls, localized at the insula, temporal gyri, amygdala, hippocampus, cingulate, and occipital gyri. Most of these statistically significant regions are in the gray matter structures, with the exception of one cluster of atrophy involved in white matter structure, i. e. , right parietal lobe white matter. Compared to previous findings documented in the literature, our findings provide a better spatial localization of the affected regions. In addition to the reduction of local volume, overall enlargement of ventricles and overall volume reduction of both white matter and gray matter are also found in XXY males.

YNIMG Journal 2004 Journal Article

Deformable registration of cortical structures via hybrid volumetric and surface warping

• Tianming Liu
• Dinggang Shen
• Christos Davatzikos

Registration of cortical structures across individuals is a very important step for quantitative analysis of the human brain cortex. This paper presents a method for deformable registration of cortical structures across individuals, using hybrid volumetric and surface warping. In the first step, a feature-based volumetric registration algorithm is used to warp a model cortical surface to the individual's space. This step greatly reduces the variation between the model and individual, thus providing a good initialization for the next step of surface warping. In the second step, a surface registration method, based on matching geometric attributes, warps the model surface to the individual. Point correspondences are also established at this step. The attribute vector, as the morphological signature of surface, was designed to be as distinctive as possible, so that each vertex on the model surface can find its correspondence on the individual surface. Experimental results on both synthesized and real brain data demonstrate the performance of the proposed method in the registration of cortical structures across individuals.

YNIMG Journal 2004 Journal Article

Measuring temporal morphological changes robustly in brain MR images via 4-dimensional template warping

• Dinggang Shen
• Christos Davatzikos

Robustly measuring subtle longitudinal brain changes is a major challenge in computational neuroanatomy. This paper describes a method for measuring temporal morphological brain changes, by means of a 4-dimensional image warping mechanism. Longitudinal stability is achieved by considering all temporal MR images of an individual simultaneously, rather than by individually warping a template to an individual, or by warping the images of one time-point to those of another time-point. Following earlier work in 3D, a local morphological signature is attached to each voxel of a sequence of images, and it includes a set of image attributes reflecting morphological characteristics of the spatiotemporal structure around the respective voxel at different scales. This attribute vector forms the basis for searching in the 4-dimensional space for a counterpart that has similar morphological signature, thereby leading to automated detection of anatomical correspondence. Ambiguities in this process are reduced by constructing attribute vectors that are highly distinctive of respective voxels, and by using a hierarchical matching procedure in which reliable and easily distinguishable voxels are used to guide the 4D deformation process. The resultant deformations are smooth both in the spatial and temporal dimensions, and are shown to significantly improve warping accuracy over a series of independent 3D warpings, in longitudinal measurements.

YNIMG Journal 2004 Journal Article

Morphological classification of brains via high-dimensional shape transformations and machine learning methods

• Zhiqiang Lao
• Dinggang Shen
• Zhong Xue
• Bilge Karacali
• Susan M. Resnick
• Christos Davatzikos

A high-dimensional shape transformation posed in a mass-preserving framework is used as a morphological signature of a brain image. Population differences with complex spatial patterns are then determined by applying a nonlinear support vector machine (SVM) pattern classification method to the morphological signatures. Significant reduction of the dimensionality of the morphological signatures is achieved via wavelet decomposition and feature reduction methods. Applying the method to MR images with simulated atrophy shows that the method can correctly detect subtle and spatially complex atrophy, even when the simulated atrophy represents only a 5% variation from the original image. Applying this method to actual MR images shows that brains can be correctly determined to be male or female with a successful classification rate of 97%, using the leave-one-out method. This proposed method also shows a high classification rate for old adults' age classification, even under difficult test scenarios. The main characteristic of the proposed methodology is that, by applying multivariate pattern classification methods, it can detect subtle and spatially complex patterns of morphological group differences which are often not detectable by voxel-based morphometric methods, because these methods analyze morphological measurements voxel-by-voxel and do not consider the entirety of the data simultaneously.

YNIMG Journal 2003 Journal Article

Very High-Resolution Morphometry Using Mass-Preserving Deformations and HAMMER Elastic Registration

• Dinggang Shen
• Christos Davatzikos

This article presents a very high-resolution voxel-based morphometric method, by using a mass-preserving deformation mechanism and a fully automated spatial normalization approach, referred to as HAMMER. By using a hierarchical attribute-based deformation strategy, HAMMER partly overcomes limitations of several existing spatial normalization methods, and it achieves a level of accuracy that makes possible morphometric measurements of spatial specificity close to the voxel dimensions. The proposed method is validated by a series of experiments, with both simulated and real brain images.

YNIMG Journal 2002 Journal Article

Measuring Size and Shape of the Hippocampus in MR Images Using a Deformable Shape Model

• Dinggang Shen
• Scott Moffat
• Susan M. Resnick
• Christos Davatzikos

A method for segmentation and quantification of the shape and size of the hippocampus is proposed, based on an automated image analysis algorithm. The algorithm uses a deformable shape model to locate the hippocampus in magnetic resonance images and to determine a geometric representation of its boundary. The deformable model combines three types of information. First, it employs information about the geometric properties of the hippocampal boundary, from a local and relatively finer scale to a more global and relatively coarser scale. Second, the model includes a statistical characterization of normal shape variation across individuals, serving as prior knowledge to the algorithm. Third, the algorithm utilizes a number of manually defined boundary points, which can help guide the model deformation to the appropriate boundaries, wherever these boundaries are weak or not clearly defined in MR images. Excellent agreement is demonstrated between the algorithm and manual segmentations by well-trained raters, with a correlation coefficient equal to 0. 97 and algorithm/rater differences statistically equivalent to interrater differences for manual definitions.