Matthew Gombolay

AAMAS Conference 2026 Conference Paper

Graph-Conditioned Diffusion for Offline Multi-Agent Reinforcement Learning

• Luis Pimentel
• Minwoo Cho
• Sean Ye
• James Ellis Grant Pagan
• Matthew Gombolay

Multi-agent reinforcement learning struggles with scalability and real-world applicability, as the high interaction variability across team compositions limits the effectiveness of online adaptive methods. Alternatively, offline RL can address these limitations by leveraging diverse offline data to facilitate learning across teams. However, existingofflineRLmethodsfailtoproducemulti-agentpolicies that can both adapt using only offline data and coordinate effectively under decentralized execution. To address these challenges, we present Graph Conditioned Diffusion (GCD), a multi-agent diffusion framework that uses graph-based communication to learn generalizable offline policies and maintain decentralization during execution. Our framework leverages the conditional generative modeling ability of diffusion models to learn multi-modal distributions of trajectories across team compositions by conditioning on team communication embeddings. We then adapt coordination online through classifier-free guidance, which steers the generative process toward behaviors that generalize across team compositions. We evaluate our method on the StarCraft II Multi-Agent Challenge v2 (SMACv2) domain, demonstrating superior generalization with an average win-rate improvement of 7. 4% to 221. 4% in unseen team compositions compared to decentralized baselines.

AAMAS Conference 2025 Conference Paper

Diverse Heterogeneous Graph Conditioned Diffusion for Multi-Agent Teaming

• Luis Pimentel
• Sean Ye
• James Ellis Grant Pagan
• Matthew Gombolay

Diverse multi-agent teams have the potential to solve complex tasks by learning effective teaming through reinforcement learning (RL). The high variability of interactions across team compositions poses scalability and real-world applicability challenges for online methods, highlighting the need for offline approaches that learn from pre-collected datasets. However, it is challenging to effectively leverage diverse data, adapt across team compositions using only offline data, and maintain decentralization during online deployment. To address these challenges, we present Heterogeneous Graph Conditioned Diffusion (HGCD), a multi-agent diffusion model that leverages the conditional generative modeling abilities of diffusion and heterogeneous multi-agent communication to learn generalizable policies offline, while ensuring decentralized execution online. We demonstrate the effectiveness of our method on StarCraft II Multi-Agent Challenge v2 (SMACv2) tasks, achieving superior generalization performance over prior state-of-the-art.

NeurIPS Conference 2025 Conference Paper

Heterogeneous Graph Transformers for Simultaneous Mobile Multi-Robot Task Allocation and Scheduling under Temporal Constraints

• Batuhan Altundas
• Shengkang Chen
• Shivika Singh
• Shivangi Deo
• Minwoo Cho
• Matthew Gombolay

Coordinating large teams of heterogeneous mobile agents to perform complex tasks efficiently has scalability bottlenecks in feasible and optimal task scheduling, with critical applications in logistics, manufacturing, and disaster response. Existing task allocation and scheduling methods, including heuristics and optimization-based solvers, often fail to scale and overlook inter-task dependencies and agent heterogeneity. We propose a novel Simultaneous Decision-Making model for Heterogeneous Multi-Agent Task Allocation and Scheduling (HM-MATAS), built on a Residual Heterogeneous Graph Transformer with edge and node-level attention. Our model encodes agent capabilities, travel times, and temporospatial constraints into a rich graph representation and is trainable via reinforcement learning. Trained on small-scale problems (10 agents, 20 tasks), our model generalizes effectively to significantly larger scenarios (up to 40 agents and 200 tasks), enabling fast, one-shot task assignment and scheduling. Our simultaneous model outperforms classical heuristics by assigning 164. 10\% more feasible tasks given temporal constraints in 3. 83\% of the time, metaheuristics by 201. 54\% in 0. 01\% of the time and exact solver by 231. 73\% in 0. 03\% of the time, while achieving $20\times$-to-$250\times$ speedup from prior graph-based methods across scales.

TMLR Journal 2024 Journal Article

Compositional Instruction Following with Language Models and Reinforcement Learning

• Vanya Cohen
• Geraud Nangue Tasse
• Nakul Gopalan
• Steven James
• Matthew Gombolay
• Ray Mooney
• Benjamin Rosman

Combining reinforcement learning with language grounding is challenging as the agent needs to explore the environment while simultaneously learning multiple language-conditioned tasks. To address this, we introduce a novel method: the compositionally-enabled reinforcement learning language agent (CERLLA). Our method reduces the sample complexity of tasks specified with language by leveraging compositional policy representations and a semantic parser trained using reinforcement learning and in-context learning. We evaluate our approach in an environment requiring function approximation and demonstrate compositional generalization to novel tasks. Our method significantly outperforms the previous best non-compositional baseline in terms of sample complexity on 162 tasks designed to test compositional generalization. Our model attains a higher success rate and learns in fewer steps than the non-compositional baseline. It reaches a success rate equal to an oracle policy's upper-bound performance of 92%. With the same number of environment steps, the baseline only reaches a success rate of 80%.

IJCAI Conference 2024 Conference Paper

Human-Robot Alignment through Interactivity and Interpretability: Don't Assume a ``Spherical Human''

• Matthew Gombolay

Interactive and interpretable robot learning can help to democratize robots, placing the power of assistive robotic systems in the hands of end-users. While machine learning-based approaches to robotics have achieved impressive results, robot learning is still a feat of costly engineering performed in controlled settings and relying upon impractical assumptions about humans. To achieve a vision in which robots can be integrated sustainably into our daily lives for robotic assistance, researchers must take a human-centered approach and develop novel approaches for human-robot alignment of robot values and behaviors. This paper amalgamates recent human factors insights and computational techniques that can support human-robot alignment through interactive and interpretable robot learning and teaming.

AAMAS Conference 2024 Conference Paper

Mixed-Initiative Human-Robot Teaming under Suboptimality with Online Bayesian Adaptation

• Manisha Natarajan
• Chunyue Xue
• Sanne van Waveren
• Karen Feigh
• Matthew Gombolay

For effective human-agent teaming, robots and other artificial intelligence (AI) agents must infer their human partner’s abilities and behavioral response patterns and adapt accordingly. Most prior works make the unrealistic assumption that one or more teammates can act near-optimally. In real-world collaboration, humans and autonomous agents can be suboptimal, especially when each only has partial domain knowledge. In this work, we develop computational modeling and optimization techniques for enhancing the performance of human-agent teams, where both the human and the robotic agent have asymmetric capabilities and act suboptimally due to incomplete environmental knowledge. We adopt an online Bayesian approach that enables a robot to infer people’s willingness to comply with its assistance in a sequential decision-making game. Our user studies show that user preferences and team performance vary with robot intervention styles, and our approach for mixed-initiative collaboration enhances objective team performance (𝑝 <. 001) and subjective measures, such as user’s trust (𝑝 <. 001) and perceived likeability of the robot (𝑝 <. 001).

NeurIPS Conference 2023 Conference Paper

Mixed-Initiative Multiagent Apprenticeship Learning for Human Training of Robot Teams

• Esmaeil Seraj
• Jerry Xiong
• Mariah Schrum
• Matthew Gombolay

Extending recent advances in Learning from Demonstration (LfD) frameworks to multi-robot settings poses critical challenges such as environment non-stationarity due to partial observability which is detrimental to the applicability of existing methods. Although prior work has shown that enabling communication among agents of a robot team can alleviate such issues, creating inter-agent communication under existing Multi-Agent LfD (MA-LfD) frameworks requires the human expert to provide demonstrations for both environment actions and communication actions, which necessitates an efficient communication strategy on a known message spaces. To address this problem, we propose Mixed-Initiative Multi-Agent Apprenticeship Learning (MixTURE). MixTURE enables robot teams to learn from a human expert-generated data a preferred policy to accomplish a collaborative task, while simultaneously learning emergent inter-agent communication to enhance team coordination. The key ingredient to MixTURE's success is automatically learning a communication policy, enhanced by a mutual-information maximizing reverse model that rationalizes the underlying expert demonstrations without the need for human generated data or an auxiliary reward function. MixTURE outperforms a variety of relevant baselines on diverse data generated by human experts in complex heterogeneous domains. MixTURE is the first MA-LfD framework to enable learning multi-robot collaborative policies directly from real human data, resulting in ~44% less human workload, and ~46% higher usability score.

AAMAS Conference 2022 Conference Paper

Learning Efficient Diverse Communication for Cooperative Heterogeneous Teaming

• Esmaeil Seraj
• Zheyuan Wang
• Rohan Paleja
• Daniel Martin
• Matthew Sklar
• Anirudh Patel
• Matthew Gombolay

High-performing teams learn intelligent and efficient communication and coordination strategies to maximize their joint utility. These teams implicitly understand the different roles of heterogeneous team members and adapt their communication protocols accordingly. Multi-Agent Reinforcement Learning (MARL) seeks to develop computational methods for synthesizing such coordination strategies, but formulating models for heterogeneous teams with different state, action, and observation spaces has remained an open problem. Without properly modeling agent heterogeneity, as in prior MARL work that leverages homogeneous graph networks, communication becomes less helpful and can even deteriorate the cooperativity and team performance. We propose Heterogeneous Policy Networks (HetNet) to learn efficient and diverse communication models for coordinating cooperative heterogeneous teams. Building on heterogeneous graph-attention networks, we show that HetNet not only facilitates learning heterogeneous collaborative policies per existing agent-class but also enables end-to-end training for learning highly efficient binarized messaging. Our empirical evaluation shows that HetNet sets a new state of the art in learning coordination and communication strategies for heterogeneous multi-agent teams by achieving an 8. 1% to 434. 7% performance improvement over the next-best baseline across multiple domains while simultaneously achieving a 200× reduction in the required communication bandwidth.

JAAMAS Journal 2022 Journal Article

Multi-UAV planning for cooperative wildfire coverage and tracking with quality-of-service guarantees

• Esmaeil Seraj
• Andrew Silva
• Matthew Gombolay

Abstract In recent years, teams of robot and Unmanned Aerial Vehicles (UAVs) have been commissioned by researchers to enable accurate, online wildfire coverage and tracking. While the majority of prior work focuses on the coordination and control of such multi-robot systems, to date, these UAV teams have not been given the ability to reason about a fire’s track (i. e. , location and propagation dynamics) to provide performance guarantee over a time horizon. Motivated by the problem of aerial wildfire monitoring, we propose a predictive framework which enables cooperation in multi-UAV teams towards collaborative field coverage and fire tracking with probabilistic performance guarantee. Our approach enables UAVs to infer the latent fire propagation dynamics for time-extended coordination in safety-critical conditions. We derive a set of novel, analytical temporal, and tracking-error bounds to enable the UAV-team to distribute their limited resources and cover the entire fire area according to the case-specific estimated states and provide a probabilistic performance guarantee. Our results are not limited to the aerial wildfire monitoring case-study and are generally applicable to problems, such as search-and-rescue, target tracking and border patrol. We evaluate our approach in simulation and provide demonstrations of the proposed framework on a physical multi-robot testbed to account for real robot dynamics and restrictions. Our quantitative evaluations validate the performance of our method accumulating \(7. 5\times\) and \(9. 0\times\) smaller tracking-error than state-of-the-art model-based and reinforcement learning benchmarks, respectively.

AAAI Conference 2021 Conference Paper

Encoding Human Domain Knowledge to Warm Start Reinforcement Learning

• Andrew Silva
• Matthew Gombolay

Deep reinforcement learning has been successful in a variety of tasks, such as game playing and robotic manipulation. However, attempting to learn tabula rasa disregards the logical structure of many domains as well as the wealth of readily available knowledge from domain experts that could help “warm start” the learning process. We present a novel reinforcement learning technique that allows for intelligent initialization of a neural network weights and architecture. Our approach permits the encoding domain knowledge directly into a neural decision tree, and improves upon that knowledge with policy gradient updates. We empirically validate our approach on two OpenAI Gym tasks and two modified StarCraft 2 tasks, showing that our novel architecture outperforms multilayer-perceptron and recurrent architectures. Our knowledge-based framework finds superior policies compared to imitation learning-based and prior knowledge-based approaches. Importantly, we demonstrate that our approach can be used by untrained humans to initially provide > 80% increase in expected reward relative to baselines prior to training (p 60% increase in expected reward after policy optimization (p = 0. 011).

AAMAS Conference 2021 Conference Paper

Multi-Agent Graph-Attention Communication and Teaming

• Yaru Niu
• Rohan Paleja
• Matthew Gombolay

High-performing teams learn effective communication strategies to judiciously share information and reduce the cost of communication overhead. Within multi-agent reinforcement learning, synthesizing effective policies requires reasoning about when to communicate, whom to communicate with, and how to process messages. We propose a novel multi-agent reinforcement learning algorithm, Multi-Agent Graph-attentIon Communication (MAGIC), with a graph-attention communication protocol in which we learn 1) a Scheduler to help with the problems of when to communicate and whom to address messages to, and 2) a Message Processor using Graph Attention Networks (GATs) with dynamic graphs to deal with communication signals. The Scheduler consists of a graph attention encoder and a differentiable attention mechanism, which outputs dynamic, differentiable graphs to the Message Processor, which enables the Scheduler and Message Processor to be trained end-to-end. We evaluate our approach on a variety of cooperative tasks, including Google Research Football. Our method outperforms baselines across all domains, achieving ≈ 10. 5% increase in reward in the most challenging domain. We also show MAGIC communicates 27. 4% more efficiently on average than baselines, is robust to stochasticity, and scales to larger state-action spaces. Finally, we demonstrate MAGIC on a physical, multi-robot testbed.

NeurIPS Conference 2021 Conference Paper

The Utility of Explainable AI in Ad Hoc Human-Machine Teaming

• Rohan Paleja
• Muyleng Ghuy
• Nadun Ranawaka Arachchige
• Reed Jensen
• Matthew Gombolay

Recent advances in machine learning have led to growing interest in Explainable AI (xAI) to enable humans to gain insight into the decision-making of machine learning models. Despite this recent interest, the utility of xAI techniques has not yet been characterized in human-machine teaming. Importantly, xAI offers the promise of enhancing team situational awareness (SA) and shared mental model development, which are the key characteristics of effective human-machine teams. Rapidly developing such mental models is especially critical in ad hoc human-machine teaming, where agents do not have a priori knowledge of others' decision-making strategies. In this paper, we present two novel human-subject experiments quantifying the benefits of deploying xAI techniques within a human-machine teaming scenario. First, we show that xAI techniques can support SA ($p<0. 05)$. Second, we examine how different SA levels induced via a collaborative AI policy abstraction affect ad hoc human-machine teaming performance. Importantly, we find that the benefits of xAI are not universal, as there is a strong dependence on the composition of the human-machine team. Novices benefit from xAI providing increased SA ($p<0. 05$) but are susceptible to cognitive overhead ($p<0. 05$). On the other hand, expert performance degrades with the addition of xAI-based support ($p<0. 05$), indicating that the cost of paying attention to the xAI outweighs the benefits obtained from being provided additional information to enhance SA. Our results demonstrate that researchers must deliberately design and deploy the right xAI techniques in the right scenario by carefully considering human-machine team composition and how the xAI method augments SA.

NeurIPS Conference 2020 Conference Paper

Interpretable and Personalized Apprenticeship Scheduling: Learning Interpretable Scheduling Policies from Heterogeneous User Demonstrations

• Rohan Paleja
• Andrew Silva
• Letian Chen
• Matthew Gombolay

Resource scheduling and coordination is an NP-hard optimization requiring an efficient allocation of agents to a set of tasks with upper- and lower bound temporal and resource constraints. Due to the large-scale and dynamic nature of resource coordination in hospitals and factories, human domain experts manually plan and adjust schedules on the fly. To perform this job, domain experts leverage heterogeneous strategies and rules-of-thumb honed over years of apprenticeship. What is critically needed is the ability to extract this domain knowledge in a heterogeneous and interpretable apprenticeship learning framework to scale beyond the power of a single human expert, a necessity in safety-critical domains. We propose a personalized and interpretable apprenticeship scheduling algorithm that infers an interpretable representation of all human task demonstrators by extracting decision-making criteria via an inferred, personalized embedding non-parametric in the number of demonstrator types. We achieve near-perfect LfD accuracy in synthetic domains and 88. 22\% accuracy on a planning domain with real-world data, outperforming baselines. Finally, our user study showed our methodology produces more interpretable and easier-to-use models than neural networks ($p < 0. 05$).

JAIR Journal 2018 Journal Article

Human-Machine Collaborative Optimization via Apprenticeship Scheduling

• Matthew Gombolay
• Reed Jensen
• Jessica Stigile
• Toni Golen
• Neel Shah
• Sung-Hyun Son
• Julie Shah

Coordinating agents to complete a set of tasks with intercoupled temporal and resource constraints is computationally challenging, yet human domain experts can solve these difficult scheduling problems using paradigms learned through years of apprenticeship. A process for manually codifying this domain knowledge within a computational framework is necessary to scale beyond the "single-expert, single-trainee" apprenticeship model. However, human domain experts often have difficulty describing their decision-making processes. We propose a new approach for capturing this decision-making process through counterfactual reasoning in pairwise comparisons. Our approach is model-free and does not require iterating through the state space. We demonstrate that this approach accurately learns multifaceted heuristics on a synthetic and real world data sets. We also demonstrate that policies learned from human scheduling demonstration via apprenticeship learning can substantially improve the efficiency of schedule optimization. We employ this human-machine collaborative optimization technique on a variant of the weapon-to-target assignment problem. We demonstrate that this technique generates optimal solutions up to 9.5 times faster than a state-of-the-art optimization algorithm.

AAAI Conference 2016 Conference Paper

Apprenticeship Scheduling for Human-Robot Teams

• Matthew Gombolay

IJCAI Conference 2016 Conference Paper

Apprenticeship Scheduling: Learning to Schedule from Human Experts

• Matthew Gombolay
• Reed Jensen
• Jessica Stigile
• Sung-Hyun Son
• Julie Shah

Coordinating agents to complete a set of tasks with intercoupled temporal and resource constraints is computationally challenging, yet human domain experts can solve these difficult scheduling problems using paradigms learned through years of apprenticeship. A process for manually codifying this domain knowledge within a computational framework is necessary to scale beyond the "single-expert, single-trainee apprenticeship model. However, human domain experts often have difficulty describing their decision-making processes, causing the codification of this knowledge to become laborious. We propose a new approach for capturing domain-expert heuristics through a pairwise ranking formulation. Our approach is model-free and does not require enumerating or iterating through a large state-space. We empirically demonstrate that this approach accurately learns multifaceted heuristics on both a synthetic data set incorporating job-shop scheduling and vehicle routing problems and a real-world data set consisting of demonstrations of experts solving a weapon-to-target assignment problem.