Author name cluster

Jiayu Chen

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

31 papers

1 author row

AAAI Conference 2026 Conference Paper

DSAP: Enhancing Generalization in Goal-Conditioned Reinforcement Learning

Yiming Wang
Kaiyan Zhao
Ming Yang
Yan Li
Furui Liu
Jiayu Chen
Leong Hou U

Goal-conditioned Reinforcement Learning (RL) is a promising direction for training agents capable of tackling a variety of tasks. However, generalizing to new goals in different environments remains a central challenge for goal-conditioned RL agents. Existing methods often rely on state abstraction, which involves learning abstracted state representations by excluding irrelevant features, to improve generalization. Despite their success in simplified settings, these methods often fail to generalize effectively to realistic environments with varied goals. In this work, we propose to enhance generalization through state abstraction from the perspective of causal inference. We hypothesize that the generalization gap arises in part due to unobserved confounders: latent variables that simultaneously influence both the global and goal states. To address this, we introduce Deconfounded State Abstraction for Policy learning (DSAP), a novel framework that mitigates backdoor confounding by employing a learned causal graph as a *proxy* for the hidden confounders. We provide theoretical analysis demonstrating that DSAP improves both the learning process and the generalization capability of goal-conditioned policies. Extensive experiments across different settings of multiple benchmarks show that our method significantly outperforms existing methods.

PDF Details DOI

AAAI Conference 2026 Short Paper

Enhancing Robustness of Offline Reinforcement Learning Under Data Corruption via Sharpness-Aware Minimization (Student Abstract)

Le Xu
Jiayu Chen

Offline reinforcement learning (RL) is vulnerable to real-world data corruption, with even robust algorithms failing under challenging observation and mixture corruptions. We posit this failure stems from data corruption creating sharp minima in the loss landscape, leading to poor generalization. To address this, we are the first to apply Sharpness-Aware Minimization (SAM) as a general-purpose, plug-and-play optimizer for offline RL. SAM seeks flatter minima, guiding models to more robust parameter regions. We integrate SAM into strong baselines for data corruption: IQL, a top-performing offline RL algorithm in this setting, and RIQL, an algorithm designed specifically for data-corruption robustness. We evaluate them on D4RL benchmarks with both random and adversarial corruption. Our SAM-enhanced methods consistently and significantly outperform the original baselines. Visualizations of the reward surface confirm that SAM finds smoother solutions, providing strong evidence for its effectiveness in improving the robustness of offline RL agents.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Explainable Oracle Bone Script Recognition via Multimodal Pictographic Reasoning

Yin Wu
Zhengxuan Zhang
Jiayu Chen
Chang Xu
Yuyu Luo
Nan Tang
Hui Xiong

Oracle Bone Script, East Asia's earliest mature writing system from over 3,500 years ago, encodes ancient cognition through visual metaphors, yet remains largely undeciphered and inaccessible, severing modern society from its cultural roots. Traditional AI methods, while accurate in classification, treat glyphs as opaque data, neglecting their pictographic essence and failing to foster public understanding—exacerbating a heritage crisis amid linguistic evolution. We pioneer a paradigm shift toward AI-driven cultural democratization, introducing OracleVis, the first human-validated multimodal dataset of glyph-image-explanation triplets, curated through expert collaborations to overcome data scarcity, bias, and incompleteness in archaeological sources. Building on this, OBS-VM, an explainability-centric multimodal large language model fine-tuned on Qwen2-VL-7B, models pictographic reasoning by balancing semantic fidelity with interpretive transparency, transforming black-box predictions into cognition-aligned narratives. Rigorous evaluations, including benchmarks and a user study with 24 non-experts, reveal our system's superiority: it outperforms GPT-4o in pictographic rationality (3.79 vs. 3.58 in human evaluation) and achieves a 35.3% relative improvement in recognition accuracy, while interactive learning boosts knowledge gains (+5.5 vs. +1.7), interest (+1.9 vs. +0.4), and confidence (+2.0 vs. +0.3) over static methods. This work illuminates AI's potential to bridge ancient wisdom and contemporary audiences, redefining heritage preservation as an inclusive, socially impactful endeavor that turns cultural alienation into enlightened engagement.

PDF Details DOI

AAAI Conference 2026 Conference Paper

Explore to Learn: Latent Exploration Through Disentangled Synergy Patterns for Reinforcement Learning in Overactuated Control

Yiming Wang
Kaiyan Zhao
Xu Li
Yan Li
Jiayu Chen
Steven Morad
Leong Hou U

Control in high-dimensional action spaces remains a fundamental challenge in reinforcement learning (RL), primarily due to inefficient exploration of the action space. While recent methods attempt to guide exploration, they often fall short of achieving the agility and coordination exhibited in biological motor control. Inspired by how organisms exploit muscle synergies for efficient movement, we propose Explore to Learn (ETL), a two-stage framework that first discovers fundamental synergy patterns and then leverages them for task-specific policy learning. In the first stage, ETL discovers underlying synergy patterns by deploying a targeted exploration policy. These patterns are modeled as latent directions in a low-dimensional space, along which the agent is guided to collect diverse and structured muscle activation trajectories. A variational autoencoder (VAE) is then trained to encode high-dimensional actions into a latent space whose dimensions correspond to the synergy patterns. In the second stage, the policy is trained entirely in this synergy-aware latent space, producing synergy coefficients that the decoder maps back to full-dimensional muscle actions. This structured representation significantly reduces the complexity of learning, while the decoder is further fine-tuned to enhance expressiveness and generalization across downstream tasks. Extensive experiments across musculoskeletal environments and the DMControl suite demonstrate that ETL consistently outperforms prior methods in both exploration efficiency and control performance, achieving superior scalability and generalization in overactuated control tasks.

PDF Details DOI

AAMAS Conference 2026 Conference Paper

ME-IGM: Individual-Global-Max in Maximum Entropy Multi-Agent Reinforcement Learning

Wentse Chen
Yuxuan Li
Shiyu Huang
Jiayu Chen
Jeff Schneider

Multi-agent credit assignment is a fundamental challenge for cooperative multi-agent reinforcement learning (MARL), where a team of agents learn from shared reward signals. The Individual-Global- Max (IGM) condition is a widely used principle for multi-agent credit assignment, requiring that the joint action determined by individual Q-functions maximizes the global Q-value. Meanwhile, the principle of maximum entropy has been leveraged to enhance exploration in MARL. However, we identify a critical limitation in existing maximum entropy MARL methods: a misalignment arises between local policies and the joint policy that maximizes the global Q-value, leading to violations of the IGM condition. To address this misalignment, we propose an order-preserving transformation. Building on it, we introduce ME-IGM, a novel maximum entropy MARL algorithm compatible with any credit assignment mechanism that satisfies the IGM condition while enjoying the benefits of maximum entropy exploration. We empirically evaluate two variants of ME-IGM: ME-QMIX and ME-QPLEX, in non-monotonic matrix games, and demonstrate their state-of-the-art performance across 17 scenarios in SMAC-v2 and Overcooked.

PDF

NeurIPS Conference 2025 Conference Paper

MALinZero: Efficient Low-Dimensional Search for Mastering Complex Multi-Agent Planning

Sizhe Tang
Jiayu Chen
Tian Lan

Monte Carlo Tree Search (MCTS), which leverages Upper Confidence Bound for Trees (UCTs) to balance exploration and exploitation through randomized sampling, is instrumental to solving complex planning problems. However, for multi-agent planning, MCTS is confronted with a large combinatorial action space that often grows exponentially with the number of agents. As a result, the branching factor of MCTS during tree expansion also increases exponentially, making it very difficult to efficiently explore and exploit during tree search. To this end, we propose MALinZero, a new approach to leverage low-dimensional representational structures on joint-action returns and enable efficient MCTS in complex multi-agent planning. Our solution can be viewed as projecting the joint-action returns into the low-dimensional space representable using a contextual linear bandit problem formulation. We solve the contextual linear bandit problem with convex and $\mu$-smooth loss functions -- in order to place more importance on better joint actions and mitigate potential representational limitations -- and derive a linear Upper Confidence Bound applied to trees (LinUCT) to enable novel multi-agent exploration and exploitation in the low-dimensional space. We analyze the regret of MALinZero for low-dimensional reward functions and propose an $(1-\tfrac1e)$-approximation algorithm for the joint action selection by maximizing a sub-modular objective. MALinZero demonstrates state-of-the-art performance on multi-agent benchmarks such as matrix games, SMAC, and SMACv2, outperforming both model-based and model-free multi-agent reinforcement learning baselines with faster learning speed and better performance.

PDF Details

NeurIPS Conference 2025 Conference Paper

Retrospective In-Context Learning for Temporal Credit Assignment with Large Language Models

Wen-Tse Chen
Jiayu Chen
Fahim Tajwar
Hao Zhu
Xintong Duan
Ruslan Salakhutdinov
Jeff Schneider

Learning from self-sampled data and sparse environmental feedback remains a fundamental challenge in training self-evolving agents. Temporal credit assignment mitigates this issue by transforming sparse feedback into dense supervision signals. However, previous approaches typically depend on domain-specific value functions for credit assignment, which suffer from poor sample efficiency and limited generalization. In this work, we propose to leverage pre-trained knowledge from large language models (LLMs) to transform sparse rewards into dense training signals (i. e. , the advantage function) through retrospective in-context learning (RICL). We further propose an online learning framework, RICOL, which iteratively refines the policy based on the credit assignment results from RICL. We empirically demonstrate that RICL can accurately estimate the advantage function with limited samples and effectively identify critical states for temporal credit assignment. Extended evaluation on the BabyAI benchmark shows that RICOL significantly improves sample efficiency compared to traditional online RL algorithms while achieving performance comparable to imitation learning from expert demonstartions. Our findings highlight the potential of leveraging LLMs for temporal credit assignment, paving the way for more sample-efficient and generalizable RL paradigms.

PDF Details

IJCAI Conference 2025 Conference Paper

Variational Offline Multi-agent Skill Discovery

Jiayu Chen
Tian Lan
Vaneet Aggarwal

Skills are effective temporal abstractions established for sequential decision making, which enable efficient hierarchical learning for long-horizon tasks and facilitate multi-task learning through their transferability. Despite extensive research, research gaps remain in multi-agent scenarios, particularly for automatically extracting subgroup coordination patterns in a multi-agent task. In this case, we propose two novel auto-encoder schemes: VO-MASD-3D and VO-MASD-Hier, to simultaneously capture subgroup- and temporal-level abstractions and form multi-agent skills, which firstly solves the aforementioned challenge. An essential algorithm component of these schemes is a dynamic grouping function that can automatically detect latent subgroups based on agent interactions in a task. Further, our method can be applied to offline multi-task data, and the discovered subgroup skills can be transferred across relevant tasks without retraining. Empirical evaluations on StarCraft tasks indicate that our approach significantly outperforms existing hierarchical multi-agent reinforcement learning (MARL) methods. Moreover, skills discovered using our method can effectively reduce the learning difficulty in MARL scenarios with delayed and sparse reward signals. The codebase is available at: https: //github. com/LucasCJYSDL/VOMASD.

PDF Details DOI

AAAI Conference 2024 Conference Paper

Accelerate Multi-Agent Reinforcement Learning in Zero-Sum Games with Subgame Curriculum Learning

Jiayu Chen
Zelai Xu
Yunfei Li
Chao Yu
Jiaming Song
Huazhong Yang
Fei Fang
Yu Wang

Learning Nash equilibrium (NE) in complex zero-sum games with multi-agent reinforcement learning (MARL) can be extremely computationally expensive. Curriculum learning is an effective way to accelerate learning, but an under-explored dimension for generating a curriculum is the difficulty-to-learn of the subgames –games induced by starting from a specific state. In this work, we present a novel subgame curriculum learning framework for zero-sum games. It adopts an adaptive initial state distribution by resetting agents to some previously visited states where they can quickly learn to improve performance. Building upon this framework, we derive a subgame selection metric that approximates the squared distance to NE values and further adopt a particle-based state sampler for subgame generation. Integrating these techniques leads to our new algorithm, Subgame Automatic Curriculum Learning (SACL), which is a realization of the subgame curriculum learning framework. SACL can be combined with any MARL algorithm such as MAPPO. Experiments in the particle-world environment and Google Research Football environment show SACL produces much stronger policies than baselines. In the challenging hide-and-seek quadrant environment, SACL produces all four emergent stages and uses only half the samples of MAPPO with self-play. The project website is at https://sites.google.com/view/sacl-neurips.

PDF Details DOI

TMLR Journal 2024 Journal Article

Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directions

Jiayu Chen
Bhargav Ganguly
Yang Xu
Yongsheng Mei
Tian Lan
Vaneet Aggarwal

Deep generative models (DGMs) have demonstrated great success across various domains, particularly in generating texts and images using models trained from offline data. Similarly, data-driven decision-making also necessitates learning a generator function from the offline data to serve as the policy. Applying DGMs in offline policy learning exhibits great potential, and numerous studies have explored in this direction. However, this field still lacks a comprehensive review and so developments of different branches are relatively independent. In this paper, we provide the first systematic review on the applications of DGMs for offline policy learning. We cover five mainstream DGMs, including Variational Auto-Encoders, Generative Adversarial Networks, Normalizing Flows, Transformers, and Diffusion Models, and their applications in both offline reinforcement learning (offline RL) and imitation learning (IL). Offline RL and IL are two main branches of offline policy learning and are widely-adopted techniques for sequential decision-making. Notably, for each type of DGM-based offline policy learning, we distill its fundamental scheme, categorize related works based on the usage of the DGM, and sort out the development process of algorithms in that field. In addition, we provide in-depth discussions on DGMs and offline policy learning as a summary, based on which we present our perspectives on future research directions. This work offers a hands-on reference for the research progress in DGMs for offline policy learning, and aims to inspire improved DGM-based offline RL or IL algorithms. For convenience, we maintain a paper list on https://github.com/LucasCJYSDL/DGMs-for-Offline-Policy-Learning.

PDF Details

TMLR Journal 2024 Journal Article

Global Convergence Guarantees for Federated Policy Gradient Methods with Adversaries

Swetha Ganesh
Jiayu Chen
Gugan Thoppe
Vaneet Aggarwal

Federated Reinforcement Learning (FRL) allows multiple agents to collaboratively build a decision making policy without sharing raw trajectories. However, if a small fraction of these agents are adversarial, it can lead to catastrophic results. We propose a policy gradient based approach that is robust to adversarial agents which can send arbitrary values to the server. Under this setting, our results form the first global convergence guarantees with general parametrization. These results demonstrate resilience with adversaries, while achieving optimal sample complexity of order $\tilde{\mathcal{O}}\left( \frac{1}{N\epsilon^2} \left( 1+ \frac{f^2}{N}\right)\right)$, where $N$ is the total number of agents and $f < N/2$ is the number of adversarial agents.

PDF Details

AAAI Conference 2024 Conference Paper

M2Doc: A Multi-Modal Fusion Approach for Document Layout Analysis

Ning Zhang
Hiuyi Cheng
Jiayu Chen
Zongyuan Jiang
Jun Huang
Yang Xue
Lianwen Jin

Document layout analysis is a crucial step for intelligent document understanding. However, many existing methods primarily focus on the visual aspects and overlook the textual features of documents. Although document pre-trained models utilize multi-modal features during the pre-training phase, they tend to operate as a unimodal pipeline when it comes to layout analysis tasks. Furthermore, current multi-modal methods perform worse than unimodal detectors on complex layout analysis datasets. To address these limitations, we propose an effective and pluggable multi-modal fusion approach named M2Doc, which fuses visual and textual features for better layout detection. M2Doc contains two pluggable multi-modal fusion modules, early-fusion and late-fusion, which align and fuse visual and textual features at the pixel level and block level. Benefitting from the concision and effectiveness of M2Doc, it can be easily applied to various detectors for better layout detection, including two-stage and end-to-end object detectors. Our experimental results demonstrate significant performance improvements in detectors equipped with M2Doc on datasets such as DocLayNet (+11.3 mAP) and M6Doc (+1.9 mAP). Furthermore, through the integration of the DINO detector with M2Doc, we achieve state-of-the-art results on DocLayNet (89.0 mAP), M6Doc (69.9 mAP), and PubLayNet (95.5 mAP). The code will be publicly released at https://github.com/johnning2333/M2Doc.

PDF Details DOI

JBHI Journal 2024 Journal Article

Reinforced Sequential Decision-Making for Sepsis Treatment: The PosNegDM Framework With Mortality Classifier and Transformer

Dipesh Tamboli
Jiayu Chen
Kiran Pranesh Jotheeswaran
Denny Yu
Vaneet Aggarwal

Sepsis, a life-threatening condition triggered by the body's exaggerated response to infection, demands urgent intervention to prevent severe complications. Existing machine learning methods for managing sepsis struggle in offline scenarios, exhibiting suboptimal performance with survival rates below 50%. This paper introduces the PosNegDM — “Reinforcement Learning with Positive and Negative Demonstrations for Sequential Decision-Making” framework utilizing an innovative transformer-based model and a feedback reinforcer to replicate expert actions while considering individual patient characteristics. A mortality classifier with 96. 7% accuracy guides treatment decisions towards positive outcomes. The PosNegDM framework significantly improves patient survival, saving 97. 39% of patients, outperforming established machine learning algorithms (Decision Transformer and Behavioral Cloning) with survival rates of 33. 4% and 43. 5%, respectively. Additionally, ablation studies underscore the critical role of the transformer-based decision maker and the integration of a mortality classifier in enhancing overall survival rates. In summary, our proposed approach presents a promising avenue for enhancing sepsis treatment outcomes, contributing to improved patient care and reduced healthcare costs.

Details DOI

NeurIPS Conference 2023 Conference Paper

A Unified Algorithm Framework for Unsupervised Discovery of Skills based on Determinantal Point Process

Jiayu Chen
Vaneet Aggarwal
Tian Lan

Learning rich skills under the option framework without supervision of external rewards is at the frontier of reinforcement learning research. Existing works mainly fall into two distinctive categories: variational option discovery that maximizes the diversity of the options through a mutual information loss (while ignoring coverage) and Laplacian-based methods that focus on improving the coverage of options by increasing connectivity of the state space (while ignoring diversity). In this paper, we show that diversity and coverage in unsupervised option discovery can indeed be unified under the same mathematical framework. To be specific, we explicitly quantify the diversity and coverage of the learned options through a novel use of Determinantal Point Process (DPP) and optimize these objectives to discover options with both superior diversity and coverage. Our proposed algorithm, ODPP, has undergone extensive evaluation on challenging tasks created with Mujoco and Atari. The results demonstrate that our algorithm outperforms state-of-the-art baselines in both diversity- and coverage-driven categories.

PDF Details

AAMAS Conference 2023 Conference Paper

Asynchronous Multi-Agent Reinforcement Learning for Efficient Real-Time Multi-Robot Cooperative Exploration

Chao Yu
Xinyi Yang
Jiaxuan Gao
Jiayu Chen
Yunfei Li
Jijia Liu
Yunfei Xiang
Ruixin Huang

We consider the problem of cooperative exploration where multiple robots need to cooperatively explore an unknown region as fast as possible. Multi-agent reinforcement learning (MARL) has recently become a trending paradigm for solving this challenge. However, existing MARL-based methods adopt action-making steps as the metric for exploration efficiency by assuming all the agents are acting in a fully synchronous manner: i. e. , every single agent produces an action simultaneously and every single action is executed instantaneously at each time step. Despite its mathematical simplicity, such a synchronous MARL formulation can be problematic for real-world robotic applications. It can be typical that different robots may take slightly different wall-clock times to accomplish an atomic action or even periodically get lost due to hardware issues. Simply waiting for every robot being ready for the next action can be particularly time-inefficient. Therefore, we propose an asynchronous MARL solution, Asynchronous Coordination Explorer (ACE), to tackle this real-world challenge. We first extend a classical MARL algorithm, multi-agent PPO (MAPPO), to the asynchronous setting and additionally apply action-delay randomization to enforce the learned policy to generalize better to varying action delays in the real world. Moreover, each navigation agent is represented as a team-size-invariant CNN-based policy, which greatly benefits real-robot deployment by handling possible robot lost and allows bandwidth-efficient intra-agent communication through low-dimensional CNN features. We first validate our approach in a grid-based scenario. Both simulation and real-robot results show that ACE reduces over 10% actual exploration time compared with classical approaches. We also apply our framework Proc. of the 22nd International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2023), A. Ricci, W. Yeoh, N. Agmon, B. An (eds.), May 29 – June 2, 2023, London, United Kingdom. © 2023 International Foundation for Autonomous Agents and Multiagent Systems (www. ifaamas. org). All rights reserved. to a high-fidelity visual-based environment, Habitat, achieving 28% improvement in exploration efficiency.

PDF

YNICL Journal 2023 Journal Article

The link between static and dynamic brain functional network connectivity and genetic risk of Alzheimer's disease

Mohammad S.E. Sendi
Elaheh Zendehrouh
Charles A. Ellis
Zening Fu
Jiayu Chen
Robyn L. Miller
Elizabeth C. Mormino
David H. Salat

Apolipoprotein E (APOE) polymorphic alleles are genetic factors associated with Alzheimer's disease (AD) risk. Although previous studies have explored the link between AD genetic risk and static functional network connectivity (sFNC), to the best of our knowledge, no previous studies have evaluated the association between dynamic FNC (dFNC) and AD genetic risk. Here, we examined the link between sFNC, dFNC, and AD genetic risk with a data-driven approach. We used rs-fMRI, demographic, and APOE data from cognitively normal individuals (N = 886) between 42 and 95 years of age (mean = 70 years). We separated individuals into low, moderate, and high-risk groups. Using Pearson correlation, we calculated sFNC across seven brain networks. We also calculated dFNC with a sliding window and Pearson correlation. The dFNC windows were partitioned into three distinct states with k-means clustering. Next, we calculated the proportion of time each subject spent in each state, called occupancy rate or OCR and frequency of visits. We compared both sFNC and dFNC features across individuals with different genetic risks and found that both sFNC and dFNC are related to AD genetic risk. We found that higher AD risk reduces within-visual sensory network (VSN) sFNC and that individuals with higher AD risk spend more time in a state with lower within-VSN dFNC. We also found that AD genetic risk affects whole-brain sFNC and dFNC in women but not men. In conclusion, we presented novel insights into the links between sFNC, dFNC, and AD genetic risk.

Details DOI

AAMAS Conference 2022 Conference Paper

Multi-agent Covering Option Discovery through Kronecker Product of Factor Graphs

Jiayu Chen
Jingdi Chen
Tian Lan
Vaneet Aggarwal

Covering option discovery has been developed to improve the exploration of reinforcement learning in single-agent scenarios with sparse reward signals, through connecting the most distant states in the embedding space provided by the Fiedler vector of the state transition graph. However, these option discovery methods cannot be directly extended to multi-agent scenarios, since the joint state space grows exponentially with the number of agents in the system. Thus, existing researches on adopting options in multi-agent scenarios still rely on single-agent option discovery and fail to directly discover the joint options that can improve the connectivity of the joint state space of agents. In this paper, we show that it is indeed possible to directly compute multi-agent options with collaborative exploratory behaviors among the agents, while still enjoying the ease of decomposition. Our key idea is to approximate the joint state space as a Kronecker graph – the Kronecker product of individual agents’ state transition graphs, based on which we can directly estimate the Fiedler vector of the joint state space using the Laplacian spectrum of individual agents’ transition graphs. This decomposition enables us to efficiently construct multi-agent joint options by encouraging agents to connect the sub-goal joint states which are corresponding to the minimum or maximum values of the estimated joint Fiedler vector. The evaluation based on multi-agent collaborative tasks shows that the proposed algorithm can successfully identify multi-agent options, and significantly outperforms prior works using single-agent options or no options, in terms of both faster exploration and higher cumulative rewards.

PDF

NeurIPS Conference 2022 Conference Paper

Scalable Multi-agent Covering Option Discovery based on Kronecker Graphs

Jiayu Chen
Jingdi Chen
Tian Lan
Vaneet Aggarwal

Covering option discovery has been developed to improve the exploration of RL in single-agent scenarios with sparse reward signals, through connecting the most distant states in the embedding space provided by the Fiedler vector of the state transition graph. Given that joint state space grows exponentially with the number of agents in multi-agent systems, existing researches still relying on single-agent option discovery either become prohibitive or fail to directly discover joint options that improve the connectivity of the joint state space. In this paper, we show how to directly compute multi-agent options with collaborative exploratory behaviors while still enjoying the ease of decomposition. Our key idea is to approximate the joint state space as a Kronecker graph, based on which we can directly estimate its Fiedler vector using the Laplacian spectrum of individual agents' transition graphs. Further, considering that directly computing the Laplacian spectrum is intractable for tasks with infinite-scale state spaces, we further propose a deep learning extension of our method by estimating eigenfunctions through NN-based representation learning techniques. The evaluation on multi-agent tasks built with simulators like Mujoco, shows that the proposed algorithm can successfully identify multi-agent options, and significantly outperforms the state-of-the-art. Codes are available at: https: //github. itap. purdue. edu/Clan-labs/Scalable MAOD via_KP.

PDF Details

NeurIPS Conference 2021 Conference Paper

Variational Automatic Curriculum Learning for Sparse-Reward Cooperative Multi-Agent Problems

Jiayu Chen
Yuanxin Zhang
Yuanfan Xu
Huimin Ma
Huazhong Yang
Jiaming Song
Yu Wang
Yi Wu

We introduce an automatic curriculum algorithm, Variational Automatic Curriculum Learning (VACL), for solving challenging goal-conditioned cooperative multi-agent reinforcement learning problems. We motivate our curriculum learning paradigm through a variational perspective, where the learning objective can be decomposed into two terms: task learning on the current curriculum, and curriculum update to a new task distribution. Local optimization over the second term suggests that the curriculum should gradually expand the training tasks from easy to hard. Our VACL algorithm implements this variational paradigm with two practical components, task expansion and entity curriculum, which produces a series of training tasks over both the task configurations as well as the number of entities in the task. Experiment results show that VACL solves a collection of sparse-reward problems with a large number of agents. Particularly, using a single desktop machine, VACL achieves 98% coverage rate with 100 agents in the simple-spread benchmark and reproduces the ramp-use behavior originally shown in OpenAI’s hide-and-seek project.

PDF Details

YNICL Journal 2020 Journal Article

NeuroMark: An automated and adaptive ICA based pipeline to identify reproducible fMRI markers of brain disorders

Yuhui Du
Zening Fu
Jing Sui
Shuang Gao
Ying Xing
Dongdong Lin
Mustafa Salman
Anees Abrol

Many mental illnesses share overlapping or similar clinical symptoms, confounding the diagnosis. It is important to systematically characterize the degree to which unique and similar changing patterns are reflective of brain disorders. Increasing sharing initiatives on neuroimaging data have provided unprecedented opportunities to study brain disorders. However, it is still an open question on replicating and translating findings across studies. Standardized approaches for capturing reproducible and comparable imaging markers are greatly needed. Here, we propose a pipeline based on the priori-driven independent component analysis, NeuroMark, which is capable of estimating brain functional network measures from functional magnetic resonance imaging (fMRI) data that can be used to link brain network abnormalities among different datasets, studies, and disorders. NeuroMark automatically estimates features adaptable to each individual subject and comparable across datasets/studies/disorders by taking advantage of the reliable brain network templates extracted from 1828 healthy controls as guidance. Four studies including 2442 subjects were conducted spanning six brain disorders (schizophrenia, autism spectrum disorder, mild cognitive impairment, Alzheimer's disease, bipolar disorder, and major depressive disorder) to evaluate validity of the proposed pipeline from different perspectives (replication of brain abnormalities, cross-study comparison, identification of subtle brain changes, and multi-disorder classification using identified biomarkers). Our results highlight that NeuroMark effectively identified replicated brain network abnormalities of schizophrenia across different datasets; revealed interesting neural clues on the overlap and specificity between autism and schizophrenia; demonstrated brain functional impairments present to varying degrees in mild cognitive impairments and Alzheimer's disease; and captured biomarkers that achieved good performance in classifying bipolar disorder and major depressive disorder.

Details DOI

YNIMG Journal 2019 Journal Article

A framework for linking resting-state chronnectome/genome features in schizophrenia: A pilot study

Barnaly Rashid
Jiayu Chen
Ishtiaque Rashid
Eswar Damaraju
Jingyu Liu
Robyn Miller
Oktay Agcaoglu
Theo G.M. van Erp

Multimodal, imaging-genomics techniques offer a platform for understanding genetic influences on brain abnormalities in psychiatric disorders. Such approaches utilize the information available from both imaging and genomics data and identify their association. Particularly for complex disorders such as schizophrenia, the relationship between imaging and genomic features may be better understood by incorporating additional information provided by advanced multimodal modeling. In this study, we propose a novel framework to combine features corresponding to functional magnetic resonance imaging (functional) and single nucleotide polymorphism (SNP) data from 61 schizophrenia (SZ) patients and 87 healthy controls (HC). In particular, the features for the functional and genetic modalities include dynamic (i. e. , time-varying) functional network connectivity (dFNC) features and the SNP data, respectively. The dFNC features are estimated from component time-courses, obtained using group independent component analysis (ICA), by computing sliding-window functional network connectivity, and then estimating subject specific states from this dFNC data using a k-means clustering approach. For each subject, both the functional (dFNC states) and SNP data are selected as features for a parallel ICA (pICA) based imaging-genomic framework. This analysis identified a significant association between a SNP component (defined by large clusters of functionally related SNPs statistically correlated with phenotype components) and time-varying or dFNC component (defined by clusters of related connectivity links among distant brain regions distributed across discrete dynamic states, and statistically correlated with genomic components) in schizophrenia. Importantly, the polygenetic risk score (PRS) for SZ (computed as a linearly weighted sum of the genotype profiles with weights derived from the odds ratios of the psychiatric genomics consortium (PGC)) was negatively correlated with the significant dFNC component, which were mostly present within a state that exhibited a lower occupancy rate in individuals with SZ compared with HC, hence identifying a potential dFNC imaging biomarker for schizophrenia. Taken together, the current findings provide preliminary evidence for a link between dFNC measures and genetic risk, suggesting the application of dFNC patterns as biomarkers in imaging genetic association study.

Details DOI

YNICL Journal 2019 Journal Article

Brain function, structure and genomic data are linked but show different sensitivity to duration of illness and disease stage in schizophrenia

Na Luo
Lin Tian
Vince D. Calhoun
Jiayu Chen
Dongdong Lin
Victor M. Vergara
Shuquan Rao
Jian Yang

The progress of schizophrenia at various stages is an intriguing question, which has been explored to some degree using single-modality brain imaging data, e.g. gray matter (GM) or functional connectivity (FC). However it remains unclear how those changes from different modalities are correlated with each other and if the sensitivity to duration of illness and disease stages across modalities is different. In this work, we jointly analyzed FC, GM volume and single nucleotide polymorphisms (SNPs) data of 159 individuals including healthy controls (HC), drug-naïve first-episode schizophrenia (FESZ) and chronic schizophrenia patients (CSZ), aiming to evaluate the links among SNP, FC and GM patterns, and their sensitivity to duration of illness and disease stages in schizophrenia. Our results suggested: 1) both GM and FC highlighted impairments in hippocampal, temporal gyrus and cerebellum in schizophrenia, which were significantly correlated with genes like SATB2, GABBR2, PDE4B, CACNA1C etc. 2) GM and FC presented gradually decrease trend (HC > FESZ>CSZ), while SNP indicated a non-gradual variation trend with un-significant group difference observed between FESZ and CSZ; 3) Group difference between HC and FESZ of FC was more remarkable than GM, and FC presented a stronger negative correlation with duration of illness than GM (p = 0.0006). Collectively, these results highlight the benefit of leveraging multimodal data and provide additional clues regarding the impact of mental illness at various disease stages.

Details DOI

YNICL Journal 2019 Journal Article

Group ICA for identifying biomarkers in schizophrenia: ‘Adaptive’ networks via spatially constrained ICA show more sensitivity to group differences than spatio-temporal regression

Mustafa S. Salman
Yuhui Du
Dongdong Lin
Zening Fu
Alex Fedorov
Eswar Damaraju
Jing Sui
Jiayu Chen

Brain functional networks identified from fMRI data can provide potential biomarkers for brain disorders. Group independent component analysis (GICA) is popular for extracting brain functional networks from multiple subjects. In GICA, different strategies exist for reconstructing subject-specific networks from the group-level networks. However, it is unknown whether these strategies have different sensitivities to group differences and abilities in distinguishing patients. Among GICA, spatio-temporal regression (STR) and spatially constrained ICA approaches such as group information guided ICA (GIG-ICA) can be used to propagate components (indicating networks) to a new subject that is not included in the original subjects. In this study, based on the same a priori network maps, we reconstructed subject-specific networks using these two methods separately from resting-state fMRI data of 151 schizophrenia patients (SZs) and 163 healthy controls (HCs). We investigated group differences in the estimated functional networks and the functional network connectivity (FNC) obtained by each method. The networks were also used as features in a cross-validated support vector machine (SVM) for classifying SZs and HCs. We selected features using different strategies to provide a comprehensive comparison between the two methods. GIG-ICA generally showed greater sensitivity in statistical analysis and better classification performance (accuracy 76.45 ± 8.9%, sensitivity 0.74 ± 0.11, specificity 0.79 ± 0.11) than STR (accuracy 67.45 ± 8.13%, sensitivity 0.65 ± 0.11, specificity 0.71 ± 0.11). Importantly, results were also consistent when applied to an independent dataset including 82 HCs and 82 SZs. Our work suggests that the functional networks estimated by GIG-ICA are more sensitive to group differences, and GIG-ICA is promising for identifying image-derived biomarkers of brain disease.

Details DOI

AAAI Conference 2018 Conference Paper

Community-Based Trip Sharing for Urban Commuting

Mohd. Hafiz Hasan
Pascal Van Hentenryck
Ceren Budak
Jiayu Chen
Chhavi Chaudhry

This paper explores Community-Based Trip Sharing which uses the structure of communities and commuting patterns to optimize car or ride sharing for urban communities. It introduces the Commuting Trip Sharing Problem (CTSP) and proposes an optimization approach to maximize trip sharing. The optimization method, which exploits trip clustering, shareability graphs, and mixed-integer programming, is applied to a dataset of 9000 daily commuting trips from a mid-size city. Experimental results show that community-based trip sharing reduces daily car usage by up to 44%, thus producing significant environmental and trafﬁc beneﬁts and reducing parking pressure. The results also indicate that daily ﬂexibility in pairing cars and passengers has signiﬁcant impact on the beneﬁts of the approach, revealing new insights on commuting patterns and trip sharing.

PDF Details

YNIMG Journal 2018 Journal Article

Dynamic functional connectivity impairments in early schizophrenia and clinical high-risk for psychosis

Yuhui Du
Susanna L. Fryer
Zening Fu
Dongdong Lin
Jing Sui
Jiayu Chen
Eswar Damaraju
Eva Mennigen

Individuals at clinical high-risk (CHR) for psychosis are characterized by attenuated psychotic symptoms. Only a minority of CHR individuals convert to full-blown psychosis. Therefore, there is a strong interest in identifying neurobiological abnormalities underlying the psychosis risk syndrome. Dynamic functional connectivity (DFC) captures time-varying connectivity over short time scales, and has the potential to reveal complex brain functional organization. Based on resting-state functional magnetic resonance imaging (fMRI) data from 70 healthy controls (HCs), 53 CHR individuals, and 58 early illness schizophrenia (ESZ) patients, we applied a novel group information guided ICA (GIG-ICA) to estimate inherent connectivity states from DFC, and then investigated group differences. We found that ESZ patients showed more aberrant connectivities and greater alterations than CHR individuals. Results also suggested that disease-related connectivity states occurred in CHR and ESZ groups. Regarding the dominant state with the highest contribution to dynamic connectivity, ESZ patients exhibited greater impairments than CHR individuals primarily in the cerebellum, frontal cortex, thalamus and temporal cortex, while CHR and ESZ populations shared common aberrances mainly in the supplementary motor area, parahippocampal gyrus and postcentral cortex. CHR-specific changes were also found in the connections between the superior frontal gyrus and calcarine cortex in the dominant state. Our findings suggest that CHR individuals generally show an intermediate functional connectivity pattern between HCs and SZ patients but also have unique connectivity alterations.

Details DOI

YNICL Journal 2018 Journal Article

Identifying functional network changing patterns in individuals at clinical high-risk for psychosis and patients with early illness schizophrenia: A group ICA study

Yuhui Du
Susanna L. Fryer
Dongdong Lin
Jing Sui
Qingbao Yu
Jiayu Chen
Barbara Stuart
Rachel L. Loewy

Although individuals at clinical high risk (CHR) for psychosis exhibit a psychosis-risk syndrome involving attenuated forms of the positive symptoms typical of schizophrenia (SZ), it remains unclear whether their resting-state brain intrinsic functional networks (INs) show attenuated or qualitatively distinct patterns of functional dysconnectivity relative to SZ patients. Based on resting-state functional magnetic imaging data from 70 healthy controls (HCs), 53 CHR individuals (among which 41 subjects were antipsychotic medication-naive), and 58 early illness SZ (ESZ) patients (among which 53 patients took antipsychotic medication) within five years of illness onset, we estimated subject-specific INs using a novel group information guided independent component analysis (GIG-ICA) and investigated group differences in INs. We found that when compared to HCs, both CHR and ESZ groups showed significant differences, primarily in default mode, salience, auditory-related, visuospatial, sensory-motor, and parietal INs. Our findings suggest that widespread INs were diversely impacted. More than 25% of voxels in the identified significant discriminative regions (obtained using all 19 possible changing patterns excepting the no-difference pattern) from six of the 15 interrogated INs exhibited monotonically decreasing Z-scores (in INs) from the HC to CHR to ESZ, and the related regions included the left lingual gyrus of two vision-related networks, the right postcentral cortex of the visuospatial network, the left thalamus region of the salience network, the left calcarine region of the fronto-occipital network and fronto-parieto-occipital network. Compared to HCs and CHR individuals, ESZ patients showed both increasing and decreasing connectivity, mainly hypo-connectivity involving 15% of the altered voxels from four INs. The left supplementary motor area from the sensory-motor network and the right inferior occipital gyrus in the vision-related network showed a common abnormality in CHR and ESZ groups. Some brain regions also showed a CHR-unique alteration (primarily the CHR-increasing connectivity). In summary, CHR individuals generally showed intermediate connectivity between HCs and ESZ patients across multiple INs, suggesting that some dysconnectivity patterns evident in ESZ predate psychosis in attenuated form during the psychosis risk stage. Hence, these connectivity measures may serve as possible biomarkers to predict schizophrenia progression.

Details DOI

YNICL Journal 2018 Journal Article

Neural correlates of cognitive function and symptoms in attention-deficit/hyperactivity disorder in adults

Kuaikuai Duan
Jiayu Chen
Vince D. Calhoun
Dongdong Lin
Wenhao Jiang
Barbara Franke
Jan K. Buitelaar
Martine Hoogman

While gray matter (GM) anomalies have been reported for attention-deficit/hyperactivity disorder (ADHD), investigating their associations with cognitive deficits and individual symptom domains can help pinpoint the neural underpinnings critical for the pathology of ADHD, particularly the persist form of ADHD. In this work, we performed both independent component analysis and voxel-based morphometry analysis on whole brain GM of 486 adults including 214 patients, 96 unaffected siblings, and 176 healthy controls, in relation to cognition and symptoms. Independent component analysis revealed that higher GM volume in inferior semilunar lobule, inferior frontal gyri, and superior and middle frontal gyri was associated with better working memory performance, and lower GM volume in cerebellar tonsil and culmen was associated with more severe inattention symptoms. Consistently, voxel-based morphometry analysis showed that higher GM volume in multiple regions of frontal lobe, cerebellum and temporal lobe was related to better working memory performance. Focusing on the networks derived from ICA, our results integrated prefrontal regions and cerebellar regions through associations with working memory and inattention symptoms, lending support for the theory of 'cool'-cognition dysfunction being mediated by inferior fronto-striato-cerebellar networks in ADHD. Siblings showed intermediate cognitive impairments between patients and controls but presented GM anomalies in unique focal regions, suggesting they are a separate group potentially affected by the shared genetic and environmental risks with ADHD patients.

Details DOI

YNIMG Journal 2017 Journal Article

Predicting individualized clinical measures by a generalized prediction framework and multimodal fusion of MRI data

Xing Meng
Rongtao Jiang
Dongdong Lin
Juan Bustillo
Thomas Jones
Jiayu Chen
Qingbao Yu
Yuhui Du

Neuroimaging techniques have greatly enhanced the understanding of neurodiversity (human brain variation across individuals) in both health and disease. The ultimate goal of using brain imaging biomarkers is to perform individualized predictions. Here we proposed a generalized framework that can predict explicit values of the targeted measures by taking advantage of joint information from multiple modalities. This framework also enables whole brain voxel-wise searching by combining multivariate techniques such as ReliefF, clustering, correlation-based feature selection and multiple regression models, which is more flexible and can achieve better prediction performance than alternative atlas-based methods. For 50 healthy controls and 47 schizophrenia patients, three kinds of features derived from resting-state fMRI (fALFF), sMRI (gray matter) and DTI (fractional anisotropy) were extracted and fed into a regression model, achieving high prediction for both cognitive scores (MCCB composite r =0. 7033, MCCB social cognition r =0. 7084) and symptomatic scores (positive and negative syndrome scale [PANSS] positive r =0. 7785, PANSS negative r =0. 7804). Moreover, the brain areas likely responsible for cognitive deficits of schizophrenia, including middle temporal gyrus, dorsolateral prefrontal cortex, striatum, cuneus and cerebellum, were located with different weights, as well as regions predicting PANSS symptoms, including thalamus, striatum and inferior parietal lobule, pinpointing the potential neuromarkers. Finally, compared to a single modality, multimodal combination achieves higher prediction accuracy and enables individualized prediction on multiple clinical measures. There is more work to be done, but the current results highlight the potential utility of multimodal brain imaging biomarkers to eventually inform clinical decision-making.

Details DOI

YNIMG Journal 2014 Journal Article

A three-way parallel ICA approach to analyze links among genetics, brain structure and brain function

Victor M. Vergara
Alvaro Ulloa
Vince D. Calhoun
David Boutte
Jiayu Chen
Jingyu Liu

Multi-modal data analysis techniques, such as the Parallel Independent Component Analysis (pICA), are essential in neuroscience, medical imaging and genetic studies. The pICA algorithm allows the simultaneous decomposition of up to two data modalities achieving better performance than separate ICA decompositions and enabling the discovery of links between modalities. However, advances in data acquisition techniques facilitate the collection of more than two data modalities from each subject. Examples of commonly measured modalities include genetic information, structural magnetic resonance imaging (MRI) and functional MRI. In order to take full advantage of the available data, this work extends the pICA approach to incorporate three modalities in one comprehensive analysis. Simulations demonstrate the three-way pICA performance in identifying pairwise links between modalities and estimating independent components which more closely resemble the true sources than components found by pICA or separate ICA analyses. In addition, the three-way pICA algorithm is applied to real experimental data obtained from a study that investigate genetic effects on alcohol dependence. Considered data modalities include functional MRI (contrast images during alcohol exposure paradigm), gray matter concentration images from structural MRI and genetic single nucleotide polymorphism (SNP). The three-way pICA approach identified links between a SNP component (pointing to brain function and mental disorder associated genes, including BDNF, GRIN2B and NRG1), a functional component related to increased activation in the precuneus area, and a gray matter component comprising part of the default mode network and the caudate. Although such findings need further verification, the simulation and in-vivo results validate the three-way pICA algorithm presented here as a useful tool in biomedical data fusion applications.

Details DOI

YNIMG Journal 2013 Journal Article

Guided exploration of genomic risk for gray matter abnormalities in schizophrenia using parallel independent component analysis with reference

Jiayu Chen
Vince D. Calhoun
Godfrey D. Pearlson
Nora Perrone-Bizzozero
Jing Sui
Jessica A. Turner
Juan R. Bustillo
Stefan Ehrlich

One application of imaging genomics is to explore genetic variants associated with brain structure and function, presenting a new means of mapping genetic influences on mental disorders. While there is growing interest in performing genome-wide searches for determinants, it remains challenging to identify genetic factors of small effect size, especially in limited sample sizes. In an attempt to address this issue, we propose to take advantage of a priori knowledge, specifically to extend parallel independent component analysis (pICA) to incorporate a reference (pICA-R), aiming to better reveal relationships between hidden factors of a particular attribute. The new approach was first evaluated on simulated data for its performance under different configurations of effect size and dimensionality. Then pICA-R was applied to a 300-participant (140 schizophrenia (SZ) patients versus 160 healthy controls) dataset consisting of structural magnetic resonance imaging (sMRI) and single nucleotide polymorphism (SNP) data. Guided by a reference SNP set derived from ANK3, a gene implicated by the Psychiatric Genomic Consortium SZ study, pICA-R identified one pair of SNP and sMRI components with a significant loading correlation of 0. 27 (p=1. 64×10−6). The sMRI component showed a significant group difference in loading parameters between patients and controls (p=1. 33×10−15), indicating SZ-related reduction in gray matter concentration in prefrontal and temporal regions. The linked SNP component also showed a group difference (p=0. 04) and was predominantly contributed to by 1030 SNPs. The effect of these top contributing SNPs was verified using association test results of the Psychiatric Genomic Consortium SZ study, where the 1030 SNPs exhibited significant SZ enrichment compared to the whole genome. In addition, pathway analyses indicated the genetic component majorly relating to neurotransmitter and nervous system signaling pathways. Given the simulation and experiment results, pICA-R may prove a promising multivariate approach for use in imaging genomics to discover reliable genetic risk factors under a scenario of relatively high dimensionality and small effect size.

Details DOI

YNIMG Journal 2012 Journal Article

Multifaceted genomic risk for brain function in schizophrenia

Jiayu Chen
Vince D. Calhoun
Godfrey D. Pearlson
Stefan Ehrlich
Jessica A. Turner
Beng-Choon Ho
Thomas H. Wassink
Andrew M. Michael

Recently, deriving candidate endophenotypes from brain imaging data has become a valuable approach to study genetic influences on schizophrenia (SZ), whose pathophysiology remains unclear. In this work we utilized a multivariate approach, parallel independent component analysis, to identify genomic risk components associated with brain function abnormalities in SZ. 5157 candidate single nucleotide polymorphisms (SNPs) were derived from genome-wide array based on their possible connections with SZ and further investigated for their associations with brain activations captured with functional magnetic resonance imaging (fMRI) during a sensorimotor task. Using data from 92 SZ patients and 116 healthy controls, we detected a significant correlation (r=0. 29; p=2. 41×10−5) between one fMRI component and one SNP component, both of which significantly differentiated patients from controls. The fMRI component mainly consisted of precentral and postcentral gyri, the major activated regions in the motor task. On average, higher activation in these regions was observed in participants with higher loadings of the linked SNP component, predominantly contributed to by 253 SNPs. 138 identified SNPs were from known coding regions of 100 unique genes. 31 identified SNPs did not differ between groups, but moderately correlated with some other group-discriminating SNPs, indicating interactions among alleles contributing toward elevated SZ susceptibility. The genes associated with the identified SNPs participated in four neurotransmitter pathways: GABA receptor signaling, dopamine receptor signaling, neuregulin signaling and glutamate receptor signaling. In summary, our work provides further evidence for the complexity of genomic risk to the functional brain abnormality in SZ and suggests a pathological role of interactions between SNPs, genes and multiple neurotransmitter pathways.

Details DOI