Arrow Research search

Author name cluster

Cheng Luo

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

34 papers
2 author rows

Possible papers

34

EAAI Journal 2026 Journal Article

A consistency-driven pseudo-labeling framework for robust functional connectivity modeling in neuropsychiatric disorder diagnosis

  • Xin Wen
  • Shijie Guo
  • Li Dong
  • Xiaobo Liu
  • Wenbo Ning
  • Jie Shi
  • Songhua Liu
  • Cheng Luo

The incidence of neuropsychiatric disorders such as Autism Spectrum Disorder (ASD), Attention Deficit Hyperactivity Disorder (ADHD), and Major Depressive Disorder (MDD) continues to rise. Deep learning-based computer-aided diagnosis (CAD) has emerged as a promising approach to alleviate the increasing burden on neuroimaging-based clinical resources. However, neuroimaging modalities such as functional magnetic resonance imaging (fMRI) involve complex spatiotemporal characteristics, making their representations susceptible to various types of noise and interference, which in turn hampers the effectiveness of CAD. To address this challenge, we propose a pseudo-label consistency-driven framework for functional connectivity (FC) reconstruction and discriminative modeling (PL-FCDM), aiming to enhance both the representational quality and discriminative power of FC features. Specifically, two complementary pseudo-labeling models are developed to independently capture discriminative features from the temporal domain (time series) and spatial domain (dynamic functional connectivity), enabling pseudo label prediction from distinct modalities. Then a consistency-based filtering strategy is applied to construct high-confidence reconstructed functional connectivity. These graphs are subsequently fed into a classification model comprising a Feature Optimization Autoencoder and a Depthwise Separable Convolutional Neural Network for efficient identification of neuropsychiatric disorders. Extensive experiments conducted on four publicly available multi-site datasets—ABIDE I, ABIDE II, ADHD-200, and REST-meta-MDD demonstrate that the proposed method achieves classification accuracies of 76. 14%, 74. 37%, 72. 89%, and 71. 15%, respectively. These results consistently outperform several state-of-the-art approaches, validating the effectiveness and robustness of the proposed framework in feature refinement and multi-disorder recognition.

YNIMG Journal 2026 Journal Article

Cortical encoding of acoustic and linguistic rhythms reflects L2 narrative comprehension

  • Jiaying Zhang
  • Junying Liang
  • Yiguang Liu
  • Cheng Luo

Speech comprehension is a multistage process involving both acoustic encoding and linguistic processing. Accumulating evidence has demonstrated that low-frequency cortical activity can track perceived linguistic units (e.g., words) on top of basic acoustic features (e.g., speech envelope). However, it remains unclear how the neural tracking of acoustic and linguistic information relates to second language (L2) speech comprehension in narrative contexts. Here, we investigate neural tracking of narrative speech for L2 listeners using electroencephalography (EEG). Notably, we introduce amplitude modulation (AM) cues aligned with word rhythm onto the basic envelope of speech and employ a frequency-tagging paradigm to measure neural responses to word and AM rhythm separately. When narrative speech was presented to L2 listeners during a speech comprehension task, reliable neural tracking of word and AM rhythm was observed in low-frequency cortical activity. While the introduction of AM cues enhances both comprehension performance and word-tracking responses, listeners with high versus low comprehension performance exhibit differences in their word-tracking responses rather than AM-tracking responses. Furthermore, the power and phase associated with word-tracking responses jointly reflect individual comprehension performance of L2 listeners. Our results indicate that bottom-up acoustic cues and top-down linguistic knowledge predominantly modulate the low-frequency neural tracking of linguistic units, which contributes to speech comprehension in a nonnative language.

EAAI Journal 2025 Journal Article

A welding sequence optimization method of multilayer thin-walled structures via combined architecture of convolutional long short-term memory-UNet and non-dominated sorting genetic algorithm II

  • Danning Fan
  • Cheng Luo
  • Yansong Zhang

Numerous welding seams in multilayer thin-walled structures of ship blocks could include thousands of welding sequences and lead to various structural deformations, significantly undermining the manufacturing quality. Welding sequence optimization based on numerical finite element (FE) simulations needs repeated model modification and calculation, facing challenges of time-consuming cost. Thus, this paper proposed a novel welding sequence optimization method based on a combined architecture of convolutional long short-term memory-UNet (ConvLSTM-UNet) and non-dominated sorting genetic algorithm Ⅱ (NSGA-Ⅱ), reducing welding deformation of multilayer thin-walled structures of ship blocks. The ConvLSTM network was used to extract the spatiotemporal characteristics of welding seams, and then welding deformation was rapidly predicted by the UNet network. NSGA-II was employed to automatically generate thousands of welding sequences, which would be input to the ConvLSTM-UNet network for fitness calculation. The multi-objective function consisted of distortion unevenness of each layer and the maximum flatness was applied for fitness evaluation and regeneration of new welding sequences. The optimized welding sequence could reduce the maximum deformation of the multilayer thin-walled structure of ship blocks up to 40. 8 %.

YNICL Journal 2025 Journal Article

Disturbed hierarchy and mediation in reward-related circuits in depression

  • Ruikun Yang
  • Junxia Chen
  • Suping Yue
  • Yue Yu
  • Jiamin Fan
  • Yuling Luo
  • Hui He
  • Mingjun Duan

BACKGROUNDS/OBJECTIVE: Deep brain stimulation (DBS) has proved the viability of alleviating depression symptoms by stimulating deep reward-related nuclei. This study aims to investigate the abnormal connectivity profiles among superficial, intermediate, and deep brain regions within the reward circuit in major depressive disorder (MDD) and therefore provides references for identifying potential superficial cortical targets for non-invasive neuromodulation. METHODS: Resting-state functional magnetic resonance imaging data were collected from a cohort of depression patients (N = 52) and demographically matched healthy controls (N = 60). Utilizing existing DBS targets as seeds, we conducted step-wise functional connectivity (sFC) analyses to delineate hierarchical pathways linking to cerebral cortices. Subsequently, the mediation effects of cortical regions on the interaction within reward-related circuits were further explored by constructing mediation models. RESULTS: In both cohorts, sFC analysis revealed two reward-related pathways from the deepest DBS targets to intermediate regions including the thalamus, insula, and anterior cingulate cortex (ACC), then to the superficial cortical cortex including medial frontal cortex, posterior default mode network (pDMN), and right dorsolateral prefrontal cortex (DLPFC). Patients exhibited reduced sFC in bilateral thalamus and medial frontal cortex in short and long steps respectively compared to healthy controls. We also discovered the disappearance of the mediation effects of superficial cortical regions on the interaction between DBS targets and intermediate regions in reward-related pathways in patients with MDD. CONCLUSION: Our findings support abnormal hierarchical connectivity and mediation effects in reward-related brain regions at different depth levels in MDD, which might elucidate the underlying pathophysiological mechanisms and inspire novel targets for non-invasive interventions.

YNIMG Journal 2025 Journal Article

Effects of antagonistic network-targeted tDCS on brain co-activation patterns depends on the networks’ electric field: a simultaneous tDCS-fMRI study

  • Hechun Li
  • Hongru Shi
  • Sisi Jiang
  • Changyue Hou
  • Haonan Pei
  • Hanxi Wu
  • María Luisa Bringas Vega
  • Gang Yao

BACKGROUND: Brain networks should be ideal targets for non-invasive brain stimulation, as network dysfunction is a common feature of various neuropsychiatric disorders. Understanding the mechanisms of network-targeted stimulation is essential for advancing its clinical applications. MATERIAL AND METHOD: The current study utilized simultaneous network-targeted transcranial direct current stimulation(tDCS) and functional magnetic resonance imaging (fMRI) to investigate the effects of tDCS targeting antagonistic networks on brain dynamics. A total of 143 healthy participants were recruited and assigned to receive central executive network (CEN)-targeted tDCS (C-targeted group), default mode network (DMN)-targeted tDCS (D-targeted group), or sham tDCS (sham group). fMRI data with three sections (pre-stimulation, during-stimulation, post-stimulation) were collected across all subjects. Individual electric field (EF) strength was simulated using individual head model. Six recurring brain patterns (co-activation patterns, CAPs) were identified. The temporal indices of these CAPs (occurrence, fraction time, persistence time) and their transition probabilities were calculated. This study first examined the effects of C-targeted / D-targeted / sham tDCS on temporal indices and further explored the contribution of brain networks' EF strength on the altered temporal indices. RESULTS: C-targeted tDCS significantly increased the temporal indices of CAPs dominated by DMN and the transition probabilities from other CAPs to DMN-dominated CAPs during stimulation. Meanwhile, the decreased temporal indices of CAP dominated by CEN, and its transition probabilities to these CAPs were also found during C-targeted tDCS. In contrast, the d-targeted tDCS had only a slight effect on brain dynamics, while sham tDCS showed no significant impact. Further fusion analyses revealed that the EF strength in the salience network made a large contribution to the temporal indices of CAPs during stimulation, highlighting tight interactions within the triple networks. Moreover, integrating the EF strength of networks with large contributions and the pre-stimulation temporal indices effectively predicted the temporal indices of CAPs during stimulation. These findings suggest that C-targeted tDCS can modulate brain dynamics and emphasize the critical role of networks' EF during stimulation. CONCLUSION: This study demonstrates the effectiveness and feasibility of network-targeted tDCS in modulating brain dynamics, providing a new choice for treating neuropsychiatric disorders characterized by aberrant brain dynamics.

NeurIPS Conference 2025 Conference Paper

OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions

  • Cheng Luo
  • Jianghui Wang
  • Bing Li
  • Siyang Song
  • Bernard Ghanem

In this paper, we introduce Online Multimodal Conversational Response Generation (OMCRG), a novel task designed to produce synchronized verbal and non-verbal listener feedback online, based on the speaker's multimodal inputs. OMCRG captures natural dyadic interactions and introduces new challenges in aligning generated audio with listeners' facial responses. To tackle these challenges, we incorporate text as an intermediate modality to connect audio and facial responses. We propose OmniResponse, a Multimodal Large Language Model (MLLM) that autoregressively generates accurate multimodal listener responses. OmniResponse leverages a pretrained LLM enhanced with two core components: Chrono-Text Markup, which precisely timestamps generated text tokens, and TempoVoice, a controllable online text-to-speech (TTS) module that outputs speech synchronized with facial responses. To advance OMCRG research, we offer ResponseNet, a dataset of 696 detailed dyadic interactions featuring synchronized split-screen videos, multichannel audio, transcripts, and annotated facial behaviors. Comprehensive evaluations on ResponseNet demonstrate that OmniResponse outperforms baseline models in terms of semantic speech content, audio-visual synchronization, and generation quality. Our dataset, code, and models are publicly available at https: //omniresponse. github. io/.

NeurIPS Conference 2025 Conference Paper

R-KV: Redundancy-aware KV Cache Compression for Reasoning Models

  • Zefan Cai
  • Wen Xiao
  • Hanshi Sun
  • Cheng Luo
  • Yikai Zhang
  • Ke Wan
  • Yucheng Li
  • Yeyang Zhou

Reasoning models have demonstrated impressive performance in self-reflection and chain-of-thought reasoning. However, they often produce excessively long outputs, leading to prohibitively large key-value (KV) caches during inference. While chain-of-thought inference significantly improves performance on complex reasoning tasks, it can also lead to reasoning failures when deployed with existing KV cache compression approaches. To address this, we propose Redundancy-aware KV Cache Compression for Reasoning models (R-KV), a novel method specifically targeting redundant tokens in reasoning models. Our method preserves nearly 100% of the full KV cache performance using only 10% of the KV cache, substantially outperforming existing KV cache baselines, which reach only 60% of the performance. Remarkably, R-KV even achieves 105% of full KV cache performance with 38% of the KV cache. This KV-cache reduction also leads to a 50% memory saving and a 2x speedup over standard chain-of-thought reasoning inference. Experimental results show that R-KV consistently outperforms existing KV cache compression baselines across two mathematical reasoning datasets.

AAAI Conference 2024 Conference Paper

Boosting Adversarial Transferability across Model Genus by Deformation-Constrained Warping

  • Qinliang Lin
  • Cheng Luo
  • Zenghao Niu
  • Xilin He
  • Weicheng Xie
  • Yuanbo Hou
  • Linlin Shen
  • Siyang Song

Adversarial examples generated by a surrogate model typically exhibit limited transferability to unknown target systems. To address this problem, many transferability enhancement approaches (e.g., input transformation and model augmentation) have been proposed. However, they show poor performances in attacking systems having different model genera from the surrogate model. In this paper, we propose a novel and generic attacking strategy, called Deformation-Constrained Warping Attack (DeCoWA), that can be effectively applied to cross model genus attack. Specifically, DeCoWA firstly augments input examples via an elastic deformation, namely Deformation-Constrained Warping (DeCoW), to obtain rich local details of the augmented input. To avoid severe distortion of global semantics led by random deformation, DeCoW further constrains the strength and direction of the warping transformation by a novel adaptive control strategy. Extensive experiments demonstrate that the transferable examples crafted by our DeCoWA on CNN surrogates can significantly hinder the performance of Transformers (and vice versa) on various tasks, including image classification, video action recognition, and audio recognition. Code is made available at https://github.com/LinQinLiang/DeCoWA.

YNIMG Journal 2024 Journal Article

Cortical encoding of hierarchical linguistic information when syllabic rhythms are obscured by echoes

  • Cheng Luo
  • Nai Ding

In speech perception, low-frequency cortical activity tracks hierarchical linguistic units (e.g., syllables, phrases, and sentences) on top of acoustic features (e.g., speech envelope). Since the fluctuation of speech envelope typically corresponds to the syllabic boundaries, one common interpretation is that the acoustic envelope underlies the extraction of discrete syllables from continuous speech for subsequent linguistic processing. However, it remains unclear whether and how cortical activity encodes linguistic information when the speech envelope does not provide acoustic correlates of syllables. To address the issue, we introduced a frequency-tagging speech stream where the syllabic rhythm was obscured by echoic envelopes and investigated neural encoding of hierarchical linguistic information using electroencephalography (EEG). When listeners attended to the echoic speech, cortical activity showed reliable tracking of syllable, phrase, and sentence levels, among which the higher-level linguistic units elicited more robust neural responses. When attention was diverted from the echoic speech, reliable neural tracking of the syllable level was also observed in contrast to deteriorated neural tracking of the phrase and sentence levels. Further analyses revealed that the envelope aligned with the syllabic rhythm could be recovered from the echoic speech through a neural adaptation model, and the reconstructed envelope yielded higher predictive power for the neural tracking responses than either the original echoic envelope or anechoic envelope. Taken together, these results suggest that neural adaptation and attentional modulation jointly contribute to neural encoding of linguistic information in distorted speech where the syllabic rhythm is obscured by echoes.

NeurIPS Conference 2024 Conference Paper

Mini-Sequence Transformers: Optimizing Intermediate Memory for Long Sequences Training

  • Cheng Luo
  • Jiawei Zhao
  • Zhuoming Chen
  • Beidi Chen
  • Anima Anandkumar

We introduce Mini-Sequence Transformer (MsT), a simple and effective methodology for highly efficient and accurate LLM training with extremely long sequences. MsT partitions input sequences and iteratively processes mini-sequences to reduce intermediate memory usage. Integrated with activation recomputation, it enables significant memory savings in both forward and backward passes. In experiments with the Llama3-8B model, with MsT, we measure no degradation in throughput or convergence even with 12x longer sequences than standard implementations. MsT is fully general, implementation-agnostic, and requires minimal code changes to integrate with existing LLM training frameworks. Integrated with the huggingface library, MsT successfully extends the maximum context length of Qwen, Mistral, and Gemma-2 by 12-24x.

YNIMG Journal 2024 Journal Article

Structural and functional alterations in MRI-negative drug-resistant epilepsy and associated gene expression features

  • Ting Liu
  • Sheng Wang
  • Yingjie Tang
  • Sisi Jiang
  • Huixia Lin
  • Fei Li
  • Dezhong Yao
  • Xian Zhu

Neuroimaging techniques have been widely used in the study of epilepsy. However, structural and functional changes in the MRI-negative drug-resistant epilepsy (DRE) and the genetic mechanisms behind the structural alterations remain poorly understood. Using structural and functional MRI, we analyzed gray matter volume (GMV) and regional homogeneity (ReHo) in DRE, drug-sensitive epilepsy (DSE) and healthy controls. Gene expression data from Allen human brain atlas and GMV/ReHo were evaluated to obtain drug resistance-related and epilepsy-associated gene expression and compared with real transcriptional data in blood. We found structural and functional alterations in the cerebellum of DRE patients, which may be related to the mechanisms of drug resistance in DRE. Our study confirms that changes in brain morphology and regional activity in DRE patients may be associated with abnormal gene expression related to nervous system development. And SP1, as an important transcription factor, plays an important role in the mechanism of drug resistance.

NeurIPS Conference 2024 Conference Paper

Towards Combating Frequency Simplicity-biased Learning for Domain Generalization

  • Xilin He
  • Jingyu Hu
  • Qinliang Lin
  • Cheng Luo
  • Weicheng Xie
  • Siyang Song
  • Muhammad Haris Khan
  • Linlin Shen

Domain generalization methods aim to learn transferable knowledge from source domains that can generalize well to unseen target domains. Recent studies show that neural networks frequently suffer from a simplicity-biased learning behavior which leads to over-reliance on specific frequency sets, namely as frequency shortcuts, instead of semantic information, resulting in poor generalization performance. Despite previous data augmentation techniques successfully enhancing generalization performances, they intend to apply more frequency shortcuts, thereby causing hallucinations of generalization improvement. In this paper, we aim to prevent such learning behavior of applying frequency shortcuts from a data-driven perspective. Given the theoretical justification of models' biased learning behavior on different spatial frequency components, which is based on the dataset frequency properties, we argue that the learning behavior on various frequency components could be manipulated by changing the dataset statistical structure in the Fourier domain. Intuitively, as frequency shortcuts are hidden in the dominant and highly dependent frequencies of dataset structure, dynamically perturbating the over-reliance frequency components could prevent the application of frequency shortcuts. To this end, we propose two effective data augmentation modules designed to collaboratively and adaptively adjust the frequency characteristic of the dataset, aiming to dynamically influence the learning behavior of the model and ultimately serving as a strategy to mitigate shortcut learning. Our code will be made publicly available.

AIIM Journal 2023 Journal Article

Osteoporosis prediction in lumbar spine X-ray images using the multi-scale weighted fusion contextual transformer network

  • Linyan Xue
  • Geng Qin
  • Shilong Chang
  • Cheng Luo
  • Ya Hou
  • Zhiyin Xia
  • Jiacheng Yuan
  • Yucheng Wang

Osteoporosis is a bone-related disease characterized by decreased bone density and mass, leading to brittle fractures. Osteoporosis assessment from radiographs using a deep learning algorithm has proven a low-cost alternative to the golden standard DXA. Due to the considerable noise and low contrast, automated diagnosis of osteoporosis in X-ray images still poses a significant challenge for traditional diagnostic methods. In this paper, an end-to-end transformer-style network was proposed, termed FCoTNet, to overcome the shortcoming of insufficient fusion of texture information and local features in the traditional CoTNet. To extract complementary geometric representations at each scale of the transformer module, we integrated parallel multi-scale feature extraction architectures in each unit layer of FCoTNet to utilize convolution to aggregate features from different receptive fields. Moreover, in order to extract small-scale texture features which were more critical to the diagnosis of osteoporosis in radiographs, larger fusion weights were assigned to the feature maps with small-size receptive fields. Afterward, the multi-scale global modeling was conducted by self-attention mechanism. The proposed model was first investigated on a private lumbar spine X-ray dataset with the 5-fold cross-validation strategy, obtaining an average accuracy of 78. 29 ± 0. 93 %, an average sensitivity of 69. 72 ± 2. 35 %, and an average specificity of 88. 92 ± 0. 67 % for the multi-classification of normal, osteopenia, and osteoporosis categories. We then conducted a controlled trial with five orthopedic clinicians to evaluate the clinical value of the model. The average clinician's accuracy improved from 61. 50 ± 10. 79 % unaided to 80. 00 ± 5. 92 % aided (18. 50 % improvement), sensitivity improved from 64. 38 ± 8. 07 % unaided to 83. 31 ± 5. 43 % aided (18. 93 % improvement), and specificity improved from 80. 11 ± 4. 72 % unaided to 89. 94 ± 3. 82 % aided (9. 83 % improvement). Meanwhile, the prediction consistency among clinicians significantly improved with the assistance of FCoTNet. Furthermore, the proposed model showed good robustness on an external test dataset. These investigations indicate that the proposed deep learning model achieves state-of-the-art performance for osteoporosis prediction, which substantially improves osteoporosis screening and reduced osteoporosis fractures.

IROS Conference 2022 Conference Paper

Improved Task Space Locomotion Controller for a Quadruped Robot with Parallel Mechanisms

  • Shunpeng Yang
  • Wenchun Lin
  • Jaeho Noh
  • Cheng Luo
  • Bill Huang
  • Wei Zhang 0013
  • Hua Chen 0007

In this work, an advanced quadruped robot with abundant kinematic loops and passive joints is introduced. Due to the existence of many closed chains, the robot dynamic model is quite complex, and is derived using the Gauss's principle of least constraint. To explicitly consider the loop-closure constraints, we propose a task-space inverse dynamics based approach to obtain the robot locomotion controller. Besides, to meet the demand of high frequency (≥ 500Hz) in controller, an alternative method is provided. It uses the projected dynamics to find an analytical mapping from the desired contact force to the desired torque of actuators under full consideration of passive joints and loop-closure constraints. The effectiveness and efficiency of the proposed algorithms in this paper have been validated by simulation with a reliable physical engine MuJoCo.

IJCAI Conference 2022 Conference Paper

Learning Multi-dimensional Edge Feature-based AU Relation Graph for Facial Action Unit Recognition

  • Cheng Luo
  • Siyang Song
  • Weicheng Xie
  • Linlin Shen
  • Hatice Gunes

The activations of Facial Action Units (AUs) mutually influence one another. While the relationship between a pair of AUs can be complex and unique, existing approaches fail to specifically and explicitly represent such cues for each pair of AUs in each facial display. This paper proposes an AU relationship modelling approach that deep learns a unique graph to explicitly describe the relationship between each pair of AUs of the target facial display. Our approach first encodes each AU's activation status and its association with other AUs into a node feature. Then, it learns a pair of multi-dimensional edge features to describe multiple task-specific relationship cues between each pair of AUs. During both node and edge feature learning, our approach also considers the influence of the unique facial display on AUs' relationship by taking the full face representation as an input. Experimental results on BP4D and DISFA datasets show that both node and edge feature learning modules provide large performance improvements for CNN and transformer-based backbones, with our best systems achieving the state-of-the-art AU recognition results. Our approach not only has a strong capability in modelling relationship cues for AU recognition but also can be easily incorporated into various backbones. Our PyTorch code is made available at https: //github. com/CVI-SZU/ME-GraphAU.

YNICL Journal 2022 Journal Article

Linking cerebellar functional gradients to transdiagnostic behavioral dimensions of psychopathology

  • Debo Dong
  • Xavier Guell
  • Sarah Genon
  • Yulin Wang
  • Ji Chen
  • Simon B. Eickhoff
  • Dezhong Yao
  • Cheng Luo

High co-morbidity and substantial overlap across psychiatric disorders encourage a transition in psychiatry research from categorical to dimensional approaches that integrate neuroscience and psychopathology. Converging evidence suggests that the cerebellum is involved in a wide range of cognitive functions and mental disorders. An important question thus centers on the extent to which cerebellar function can be linked to transdiagnostic dimensions of psychopathology. To address this question, we used a multivariate data-driven statistical technique (partial least squares) to identify latent dimensions linking human cerebellar connectome as assessed by functional MRI to a large set of clinical, cognitive, and trait measures across 198 participants, including healthy controls (n = 92) as well as patients diagnosed with attention-deficit/hyperactivity disorder (n = 35), bipolar disorder (n = 36), and schizophrenia (n = 35). Macroscale spatial gradients of connectivity at voxel level were used to characterize cerebellar connectome properties, which provide a low-dimensional representation of cerebellar connectivity, i.e., a sensorimotor-supramodal hierarchical organization. This multivariate analysis revealed significant correlated patterns of cerebellar connectivity gradients and behavioral measures that could be represented into four latent dimensions: general psychopathology, impulsivity and mood, internalizing symptoms and executive dysfunction. Each dimension was associated with a unique spatial pattern of cerebellar connectivity gradients across all participants. Multiple control analyses and 10-fold cross-validation confirmed the robustness and generalizability of the yielded four dimensions. These findings highlight the relevance of cerebellar connectivity as a necessity for the study and classification of transdiagnostic dimensions of psychopathology and call on researcher to pay more attention to the role of cerebellum in the dimensions of psychopathology, not just within the cerebral cortex.

YNIMG Journal 2022 Journal Article

Working memory asymmetrically modulates auditory and linguistic processing of speech

  • Yiguang Liu
  • Cheng Luo
  • Jing Zheng
  • Junying Liang
  • Nai Ding

Working memory load can modulate speech perception. However, since speech perception and working memory are both complex functions, it remains elusive how each component of the working memory system interacts with each speech processing stage. To investigate this issue, we concurrently measure how the working memory load modulates neural activity tracking three levels of linguistic units, i.e., syllables, phrases, and sentences, using a multiscale frequency-tagging approach. Participants engage in a sentence comprehension task and the working memory load is manipulated by asking them to memorize either auditory verbal sequences or visual patterns. It is found that verbal and visual working memory load modulate speech processing in similar manners: Higher working memory load attenuates neural activity tracking of phrases and sentences but enhances neural activity tracking of syllables. Since verbal and visual WM load similarly influence the neural responses to speech, such influences may derive from the domain-general component of WM system. More importantly, working memory load asymmetrically modulates lower-level auditory encoding and higher-level linguistic processing of speech, possibly reflecting reallocation of attention induced by mnemonic load.

YNICL Journal 2021 Journal Article

Structural and functional reorganization of contralateral hippocampus after temporal lobe epilepsy surgery

  • Wei Li
  • Yuchao Jiang
  • Yingjie Qin
  • Baiwan Zhou
  • Du Lei
  • Heng Zhang
  • Ding Lei
  • Dezhong Yao

OBJECTIVE: To explore the structural and functional reorganization of contralateral hippocampus in patients with unilateral mesial temporal lobe epilepsy (mTLE) who achieved seizure-freedom after anterior temporal lobectomy (ATL). METHODS: We obtained high-resolution structural MRI and resting-state functional MRI data in 28 unilateral mTLE patients and 29 healthy controls. Patients were scanned before and three and 24 months after surgery while controls were scanned only once. Hippocampal gray matter volume (GMV) and functional connectivity (FC) were assessed. RESULTS: No obvious GMV changes were observed in contralateral hippocampus before and after successful surgery. Before surgery, ipsilateral hippocampus showed increased FC with ipsilateral insula (INS) and temporoparietal junction (TPJ), but decreased FC with widespread bilateral regions, as well as contralateral hippocampus. After successful ATL, contralateral hippocampus showed: (1) decreased FC with ipsilateral INS at three months follow-up, without further changes; (2) decreased FC with ipsilateral TPJ, postcentral gyrus and rolandic operculum at three months, with an obvious increase at 24 months follow-up; (3) increased FC with bilateral medial prefrontal cortex (MPFC) and superior frontal gyrus (SFG) at three months follow-up, without further changes. CONCLUSIONS: Successful ATL may not lead to an obvious structural reorganization in contralateral hippocampus. Surgical manipulation may lead to a transient FC reduction of contralateral hippocampus. Increased FC between contralateral hippocampus and bilateral MPFC and SFG may be related to postoperative functional remodeling.

YNIMG Journal 2020 Journal Article

Objects seen as scenes: Neural circuitry for attending whole or parts

  • Mitchell Valdés-Sosa
  • Marlis Ontivero-Ortega
  • Jorge Iglesias-Fuster
  • Agustin Lage-Castellanos
  • Jinnan Gong
  • Cheng Luo
  • Ana Maria Castro-Laguardia
  • Maria Antonieta Bobes

Depending on our goals, we pay attention to the global shape of an object or to the local shape of its parts, since it’s difficult to do both at once. This typically effortless process can be impaired in disease. However, it is not clear which cortical regions carry the information needed to constrain shape processing to a chosen global/local level. Here, novel stimuli were used to dissociate functional MRI responses to global and local shapes. This allowed identification of cortical regions containing information about level (independent from shape). Crucially, these regions overlapped part of the cortical network implicated in scene processing. As expected, shape information (independent of level) was mainly located in category-selective areas specialized for object- and face-processing. Regions with the same informational profile were strongly linked (as measured by functional connectivity), but were weak when the profiles diverged. Specifically, in the ventral-temporal-cortex (VTC) regions favoring level and shape were consistently separated by the mid-fusiform sulcus (MFS). These regions also had limited crosstalk despite their spatial proximity, thus defining two functional pathways within VTC. We hypothesize that object hierarchical level is processed by neural circuitry that also analyses spatial layout in scenes, contributing to the control of the spatial-scale used for shape recognition. Use of level information tolerant to shape changes could guide whole/part attentional selection but facilitate illusory shape/level conjunctions under impoverished vision.

YNIMG Journal 2020 Journal Article

Visual target detection in a distracting background relies on neural encoding of both visual targets and background

  • Cheng Luo
  • Nai Ding

The ability to detect visual targets in complex background varies across individuals and are affected by factors such as stimulus saliency and top-down attention. Here, we investigated how the saliency of visual background (naturalistic cartoon video vs. blank screen) and top-down attention (single vs. dual tasks) separately affect individual ability to detect visual targets. Behaviorally, we found that target detection accuracy decreased and reaction time elongated when the background was salient or during dual tasking. The EEG response to visual background was recorded using a novel stimulus tagging technique. This response was strongest in occipital electrodes and was sensitive to background saliency but not dual tasking. In contrast, the event-related potential (ERP) evoked by the visual target was strongest in central electrodes, and was affected by both background saliency and dual tasking. With a cartoon background, the EEG responses to visual targets, presented in the central visual field, and the EEG responses to peripheral visual background could both predict individual target detection performance. When these two responses were combined, better prediction was achieved. These results suggest that neural processing of visual targets and background jointly contribute to individual visual target detection performance.

YNIMG Journal 2019 Journal Article

Auditory and language contributions to neural encoding of speech features in noisy environments

  • Jiajie Zou
  • Jun Feng
  • Tianyong Xu
  • Peiqing Jin
  • Cheng Luo
  • Jianfeng Zhang
  • Xunyi Pan
  • Feiyan Chen

Recognizing speech in noisy environments is a challenging task that involves both auditory and language mechanisms. Previous studies have demonstrated human auditory cortex can reliably track the temporal envelope of speech in noisy environments, which provides a plausible neural basis for noise-robust speech recognition. The current study aimed at teasing apart auditory and language contributions to noise-robust envelope tracking by comparing the neural responses of 2 groups of listeners, i. e. , native listeners and foreign listeners who did not understand the testing language. In the experiment, speech signals were mixed with spectrally matched stationary noise at 4 intensity levels and listeners’ neural responses were recorded using electroencephalography (EEG). When the noise intensity increased, the neural response gain increased in both groups of listeners, demonstrating auditory gain control. Language comprehension generally reduced the response gain and envelope-tracking precision, and modulated the spatial and temporal profile of envelope-tracking activity. Based on the spatio-temporal dynamics of envelope-tracking activity, a linear classifier can jointly decode the 2 listener groups and 4 levels of noise intensity. Altogether, the results showed that without feedback from language processing, auditory mechanisms such as gain control can lead to a noise-robust speech representation. High-level language processing modulated the spatio-temporal profile of the neural representation of speech envelope, instead of generally enhancing the envelope representation.

YNICL Journal 2019 Journal Article

BOLD-fMRI activity informed by network variation of scalp EEG in juvenile myoclonic epilepsy

  • Yun Qin
  • Sisi Jiang
  • Qiqi Zhang
  • Li Dong
  • Xiaoyan Jia
  • Hui He
  • Yutong Yao
  • Huanghao Yang

Epilepsy is marked by hypersynchronous bursts of neuronal activity, and seizures can propagate variably to any and all areas, leading to brain network dynamic organization. However, the relationship between the network characteristics of scalp EEG and blood oxygenation level-dependent (BOLD) responses in epilepsy patients is still not well known. In this study, simultaneous EEG and fMRI data were acquired in 18 juvenile myoclonic epilepsy (JME) patients. Then, the adapted directed transfer function (ADTF) values between EEG electrodes were calculated to define the time-varying network. The variation of network information flow within sliding windows was used as a temporal regressor in fMRI analysis to predict the BOLD response. To investigate the EEG-dependent functional coupling among the responding regions, modulatory interactions were analyzed for network variation of scalp EEG and BOLD time courses. The results showed that BOLD activations associated with high network variation were mainly located in the thalamus, cerebellum, precuneus, inferior temporal lobe and sensorimotor-related areas, including the middle cingulate cortex (MCC), supplemental motor area (SMA), and paracentral lobule. BOLD deactivations associated with medium network variation were found in the frontal, parietal, and occipital areas. In addition, modulatory interaction analysis demonstrated predominantly directional negative modulation effects among the thalamus, cerebellum, frontal and sensorimotor-related areas. This study described a novel method to link BOLD response with simultaneous functional network organization of scalp EEG. These findings suggested the validity of predicting epileptic activity using functional connectivity variation between electrodes. The functional coupling among the thalamus, frontal regions, cerebellum and sensorimotor-related regions may be characteristically involved in epilepsy generation and propagation, which provides new insight into the pathophysiological mechanisms and intervene targets for JME.

YNIMG Journal 2019 Journal Article

BOLD-fMRI reveals the association between renal oxygenation and functional connectivity in the aging brain

  • Hechun Li
  • Weifang Cao
  • Xingxing Zhang
  • Bo Sun
  • Sisi Jiang
  • Jianfu Li
  • Chang Liu
  • Wenjie Yin

Aging is accompanied by a decline in physical and cognitive function. Vascular aging may provide a major influence on these measures. The purpose of this study was to explore the relationship between renal oxygenation and functional connectivity of the aging brain because of the anatomic and hemodynamic similarities between cerebral and renal vessels. Fifty-two healthy older adults were recruited to undergo a BOLD-fMRI scan of the brain and kidneys, and forty-four healthy younger subjects were recruited as the control group. First, cerebral functional connectivity density (FCD) was used to evaluate functional connectivity. Renal medullary and cortical R2* values were extracted respectively, and the ratio of medullary and cortical R2* values (MCR) was calculated. Then, the association between brain FCD and renal MCR was analyzed. Compared with younger adults, the elderly group showed higher renal medullary R2* and MCR, which might reflect a slight abnormality of renal oxygenation with aging. The older subjects also showed enhanced FCD in bilateral motor-related regions and decreased FCD in regions of the default mode network (DMN). The findings indicated that the functional connectivity in the DMN and motor cortices was vulnerable to aging. Moreover, the altered brain FCD values in the watershed regions, DMN and motor cortices were significantly correlated with the renal MCR value in the elderly group. The association between renal oxygenation abnormalities and spontaneous activity in the brain might reflect vascular aging and its influence on the kidney and brain during aging to some extent. This study provided a new perspective for understanding the relationship between tissue oxygenation and brain functional connectivity.

YNICL Journal 2019 Journal Article

Common increased hippocampal volume but specific changes in functional connectivity in schizophrenia patients in remission and non-remission following electroconvulsive therapy: A preliminary study

  • Yuchao Jiang
  • Lihua Xu
  • Xiangkui Li
  • Yingying Tang
  • Pingfu Wang
  • Chunbo Li
  • Dezhong Yao
  • Jijun Wang

Electroconvulsive therapy (ECT) is considered a treatment option in patients with drug-resistant schizophrenia (SZ). However, approximately one-third of patients do not benefit from ECT in the clinic. Thus, it is critical to investigate differences between ECT responders and non-responders. Accumulated evidence has indicated that one region of ECT action is the hippocampus, which also plays an important role in SZ pathophysiology. To date, no studies have investigated differences in ECT effects in the hippocampus between treatment responders and non-responders. This study recruited twenty-one SZ patients treated for four weeks with ECT (MSZ, n = 21) and twenty-one SZ patients who received pharmaceutical therapy (DSZ, n = 21). The MSZ group was further categorized into responders (MSR, n = 10) or non-responders (MNR, n = 11) based on treatment outcomes by the criterion of a 50% reduction in the Positive and Negative Syndrome Scale total scores. Using structural and resting-state functional MRI, we measured the hippocampal volume and functional connectivity (FC) in all SZ patients (before and after treatment) and 23 healthy controls. In contrast to pharmaceutical therapy, ECT induced bilateral hippocampal volume increases in the MSZ. Both the MSR and MNR exhibited hippocampal expansion after ECT, whereas a lower baseline volume in one of hippocampal subfield (hippocampus-amygdala transition area) was found in the MNR. After ECT, increased FC between the hippocampus and brain networks associated with cognitive function was only observed in the MSR. The mechanism of action of ECT in schizophrenia is complex. A combination of baseline impairment level, ECT-introduced morphological changes and post-ECT FC increases in the hippocampus may jointly contribute to the post-ECT symptom improvements in patients with SZ.

YNICL Journal 2019 Journal Article

Different patterns of white matter changes after successful surgery of mesial temporal lobe epilepsy

  • Wei Li
  • Dongmei An
  • Xin Tong
  • Wenyu Liu
  • Fenglai Xiao
  • Jiechuan Ren
  • Running Niu
  • Yingying Tang

OBJECTIVES: To explore the dynamic changes of white matters following anterior temporal lobectomy (ATL) in mesial temporal lobe epilepsy (MTLE) patients who achieved seizure-free at two-year follow-up. METHODS: Diffusion tensor imaging (DTI) was obtained in ten MTLE patients at five serial time points: before surgery, three months, six months, 12 months and 24 months after surgery, as well as in 11 age- and sex-matched healthy controls at one time point. Regions with significant postoperative fractional anisotropy (FA) changes and their dynamic changes were confirmed by comparing all preoperative and postoperative data using Tract-Based Spatial Statistics (TBSS). RESULTS: After successful ATL, significant FA changes were found in widespread ipsilateral and contralateral white matter regions (P <.05, FWE correction). Ipsilateral external capsule, cingulum, superior corona radiate, body of corpus callosum, inferior longitudinal fasciculus, optic radiation and contralateral inferior cerebellar peduncle, inferior longitudinal fasciculus showed significant FA decrease at three months after surgery, without further changes. Ipsilateral superior cerebellar peduncle and contralateral corpus callosum, anterior corona radiate, external capsule, optic radiation showed significant FA decrease at three months follow up but increase later. Ipsilateral cerebral peduncle and contralateral middle cerebellar peduncle showed significant FA decrease at three months follow up, with further decrease after that. While ipsilateral posterior limb of internal capsule, retrolenticular part of internal capsule and contralateral posterior corona radiate showed significant FA increase after surgery. CONCLUSIONS: FA changes after successful ATL presented as four distinct patterns, reflecting different structural adaptions following epilepsy surgery. Some FA increases indicated the reversibility of preoperative diffusion abnormalities and the possibility of structural reorganization, especially in the contralateral hemisphere.

YNICL Journal 2019 Journal Article

Low-rank network signatures in the triple network separate schizophrenia and major depressive disorder

  • Wei Han
  • Christian Sorg
  • Changgang Zheng
  • Qinli Yang
  • Xiaosong Zhang
  • Arvid Ternblom
  • Cobbinah Bernard Mawuli
  • Lianli Gao

Brain imaging studies have revealed that functional and structural brain connectivity in the so-called triple network (i.e., default mode network (DMN), salience network (SN) and central executive network (CEN)) are consistently altered in schizophrenia. However, similar changes have also been found in patients with major depressive disorder, prompting the question of specific triple network signatures for the two disorders. In this study, we proposed Supervised Convex Nonnegative Matrix Factorization (SCNMF) to extract distributed multi-modal brain patterns. These patterns distinguish schizophrenia and major depressive disorder in a latent low-dimensional space of the triple brain network. Specifically, 21 patients of schizophrenia and 25 patients of major depressive disorder were assessed by T1-weighted, diffusion-weighted, and resting-state functional MRIs. Individual structural and functional connectivity networks, based on pre-defined regions of the triple network were constructed, respectively. Afterwards, SCNMF was employed to extract the discriminative patterns. Experiments indicate that SCNMF allows extracting the low-rank discriminative patterns between the two disorders, achieving a classification accuracy of 82.6% based on the extracted functional and structural abnormalities with support vector machine. Experimental results show the specific brain patterns for schizophrenia and major depressive disorder that are multi-modal, complex, and distributed in the triple network. Parts of the prefrontal cortex including superior frontal gyri showed variation between patients with schizophrenia and major depression due to structural properties. In terms of functional properties, the middle cingulate cortex, inferior parietal lobule, and cingulate cortex were the most discriminative regions.

YNIMG Journal 2019 Journal Article

White-matter functional networks changes in patients with schizophrenia

  • Yuchao Jiang
  • Cheng Luo
  • Xuan Li
  • Yingjia Li
  • Hang Yang
  • Jianfu Li
  • Xin Chang
  • Hechun Li

Resting-state functional MRI (rsfMRI) is a useful technique for investigating the functional organization of human gray-matter in neuroscience and neuropsychiatry. Nevertheless, most studies have demonstrated the functional connectivity and/or task-related functional activity in the gray-matter. White-matter functional networks have been investigated in healthy subjects. Schizophrenia has been hypothesized to be a brain disorder involving insufficient or ineffective communication associated with white-matter abnormalities. However, previous studies have mainly examined the structural architecture of white-matter using MRI or diffusion tensor imaging and failed to uncover any dysfunctional connectivity within the white-matter on rsfMRI. The current study used rsfMRI to evaluate white-matter functional connectivity in a large cohort of ninety-seven schizophrenia patients and 126 healthy controls. Ten large-scale white-matter networks were identified by a cluster analysis of voxel-based white-matter functional connectivity and classified into superficial, middle and deep layers of networks. Evaluation of the spontaneous oscillation of white-matter networks and the functional connectivity between them showed that patients with schizophrenia had decreased amplitudes of low-frequency oscillation and increased functional connectivity in the superficial perception-motor networks. Additionally, we examined the interactions between white-matter and gray-matter networks. The superficial perception-motor white-matter network had decreased functional connectivity with the cortical perception-motor gray-matter networks. In contrast, the middle and deep white-matter networks had increased functional connectivity with the superficial perception-motor white-matter network and the cortical perception-motor gray-matter network. Thus, we presumed that the disrupted association between the gray-matter and white-matter networks in the perception-motor system may be compensated for through the middle-deep white-matter networks, which may be the foundation of the extensively disrupted connections in schizophrenia.

IJCAI Conference 2018 Conference Paper

Your Tweets Reveal What You Like: Introducing Cross-media Content Information into Multi-domain Recommendation

  • Weizhi Ma
  • Min Zhang
  • Chenyang Wang
  • Cheng Luo
  • Yiqun Liu
  • Shaoping Ma

Cold start is a challenging problem in recommender systems. Many previous studies attempt to utilize extra information from other platforms to alleviate the problem. Most of the leveraged information is on-topic, directly related to users' preferences in the target domain. Thought to be unrelated, users' off-topic content information (such as user tweets) is usually omitted. However, the off-topic content information also helps to indicate the similarity of users on their tastes, interests, and opinions, which matches the underlying assumption of Collaborative Filtering (CF) algorithms. In this paper, we propose a framework to capture the features from user's off-topic content information in social media and introduce them into Matrix Factorization (MF) based algorithms. The framework is easy to understand and flexible in different embedding approaches and MF based algorithms. To the best of our knowledge, there is no previous study in which user's off-topic content in other platforms is taken into consideration. By capturing the cross-platform content including both on-topic and off-topic information, multiple algorithms with several embedding learning approaches have achieved significant improvements in rating prediction on three datasets. Especially in cold start scenarios, we observe greater enhancement. The results confirm our suggestion that off-topic cross-media information also contributes to the recommendation.

YNIMG Journal 2016 Journal Article

Structural and functional correlates of motor imagery BCI performance: Insights from the patterns of fronto-parietal attention network

  • Tao Zhang
  • Tiejun Liu
  • Fali Li
  • Mengchen Li
  • Dongbo Liu
  • Rui Zhang
  • Hui He
  • Peiyang Li

Motor imagery (MI)-based brain-computer interfaces (BCIs) have been widely used for rehabilitation of motor abilities and prosthesis control for patients with motor impairments. However, MI-BCI performance exhibits a wide variability across subjects, and the underlying neural mechanism remains unclear. Several studies have demonstrated that both the fronto-parietal attention network (FPAN) and MI are involved in high-level cognitive processes that are crucial for the control of BCIs. Therefore, we hypothesized that the FPAN may play an important role in MI-BCI performance. In our study, we recorded multi-modal datasets consisting of MI electroencephalography (EEG) signals, T1-weighted structural and resting-state functional MRI data for each subject. MI-BCI performance was evaluated using the common spatial pattern to extract the MI features from EEG signals. One cortical structural feature (cortical thickness (CT)) and two measurements (degree centrality (DC) and eigenvector centrality (EC)) of node centrality were derived from the structural and functional MRI data, respectively. Based on the information extracted from the EEG and MRI, a correlation analysis was used to elucidate the relationships between the FPAN and MI-BCI performance. Our results show that the DC of the right ventral intraparietal sulcus, the EC and CT of the left inferior parietal lobe, and the CT of the right dorsolateral prefrontal cortex were significantly associated with MI-BCI performance. Moreover, the receiver operating characteristic analysis and machine learning classification revealed that the EC and CT of the left IPL could effectively predict the low-aptitude BCI users from the high-aptitude BCI users with 83. 3% accuracy. Those findings consistently reveal that the individuals who have efficient FPAN would perform better on MI-BCI. Our findings may deepen the understanding of individual variability in MI-BCI performance, and also may provide a new biomarker to predict individual MI-BCI performance.

YNIMG Journal 2015 Journal Article

Characterizing nonlinear relationships in functional imaging data using eigenspace maximal information canonical correlation analysis (emiCCA)

  • Li Dong
  • Yangsong Zhang
  • Rui Zhang
  • Xingxing Zhang
  • Diankun Gong
  • Pedro A. Valdes-Sosa
  • Peng Xu
  • Cheng Luo

Many important problems in the analysis of neuroimages can be formulated as discovering the relationship between two sets of variables, a task for which linear techniques such as canonical correlation analysis (CCA) have been commonly used. However, to further explore potential nonlinear processes that might co-exist with linear ones in brain function, a more flexible method is required. Here, we propose a new unsupervised and data-driven method, termed the eigenspace maximal information canonical correlation analysis (emiCCA), which is capable of automatically capturing the linear and/or nonlinear relationships between various data sets. A simulation confirmed the superior performance of emiCCA in comparison with linear CCA and kernel CCA (a nonlinear version of CCA). An emiCCA framework for functional magnetic resonance imaging (fMRI) data processing was designed and applied to data from a real motor execution fMRI experiment. This analysis uncovered one linear (in primary motor cortex) and a few nonlinear networks (e. g. , in the supplementary motor area, bilateral insula, and cerebellum). This suggests that these various task-related brain areas are part of networks that also contribute to the execution of movements of the hand. These results suggest that emiCCA is a promising technique for exploring various data.

YNICL Journal 2014 Journal Article

Patient-specific connectivity pattern of epileptic network in frontal lobe epilepsy

  • Cheng Luo
  • Dongmei An
  • Dezhong Yao
  • Jean Gotman

There is evidence that focal epilepsy may involve the dysfunction of a brain network in addition to the focal region. To delineate the characteristics of this epileptic network, we collected EEG/fMRI data from 23 patients with frontal lobe epilepsy. For each patient, EEG/fMRI analysis was first performed to determine the BOLD response to epileptic spikes. The maximum activation cluster in the frontal lobe was then chosen as the seed to identify the epileptic network in fMRI data. Functional connectivity analysis seeded at the same region was also performed in 63 healthy control subjects. Nine features were used to evaluate the differences of epileptic network patterns in three connection levels between patients and controls. Compared with control subjects, patients showed overall more functional connections between the epileptogenic region and the rest of the brain and higher laterality. However, the significantly increased connections were located in the neighborhood of the seed, but the connections between the seed and remote regions actually decreased. Comparing fMRI runs with interictal epileptic discharges (IEDs) and without IEDs, the patient-specific connectivity pattern was not changed significantly. These findings regarding patient-specific connectivity patterns of epileptic networks in FLE reflect local high connectivity and connections with distant regions differing from those of healthy controls. Moreover, the difference between the two groups in most features was observed in the strictest of the three connection levels. The abnormally high connectivity might reflect a predominant attribute of the epileptic network, which may facilitate propagation of epileptic activity among regions in the network.

YNIMG Journal 2014 Journal Article

Simultaneous EEG-fMRI: Trial level spatio-temporal fusion for hierarchically reliable information discovery

  • Li Dong
  • Diankun Gong
  • Pedro A. Valdes-Sosa
  • Yang Xia
  • Cheng Luo
  • Peng Xu
  • Dezhong Yao

Simultaneous electroencephalography (EEG) and functional magnetic resonance imaging (fMRI) have been pursued in an effort to integrate complementary noninvasive information on brain activity. The primary goal involves better information discovery of the event-related neural activations at a spatial region of the BOLD fluctuation with the temporal resolution of the electrical signal. Many techniques and algorithms have been developed to integrate EEGs and fMRIs; however, the relative reliability of the integrated information is unclear. In this work, we propose a hierarchical framework to ensure the relative reliability of the integrated results and attempt to understand brain activation using this hierarchical ideal. First, spatial Independent Component Analysis (ICA) of fMRI and temporal ICA of EEG were performed to extract features at the trial level. Second, the maximal information coefficient (MIC) was adopted to temporally match them across the modalities for both linear and non-linear associations. Third, fMRI-constrained EEG source imaging was utilized to spatially match components across modalities. The simultaneously occurring events in the above two match steps provided EEG-fMRI spatial–temporal reliable integrated information, resulting in the most reliable components with high spatial and temporal resolution information. The other components discovered in the second or third steps provided second-level complementary information for flexible and cautious explanations. This paper contains two simulations and an example of real data, and the results indicate that the framework is a feasible approach to reveal cognitive processing in the human brain.

v2026.09.13