Arrow Research search

Author name cluster

Guillaume Sartoretti

Possible papers associated with this exact author name in Arrow. This page groups case-insensitive exact name matches and is not a full identity disambiguation profile.

35 papers
2 author rows

Possible papers

35

AAAI Conference 2026 Conference Paper

Social Behavior as a Key to Learning-Based Multi-Agent Pathfinding Dilemmas (Abstract Reprint)

  • Chengyang He
  • Tanishq Duhan
  • Parth Tulsyan
  • Patrick Kim
  • Guillaume Sartoretti

The Multi-agent Path Finding (MAPF) problem involves finding collision-free paths for a team of agents in a known, static environment, with important applications in warehouse automation, logistics, or last-mile delivery. To meet the needs of these large-scale applications, current learning-based methods often deploy the same fully trained, decentralized network to all agents to improve scalability. However, such parameter sharing typically results in homogeneous behaviors among agents, which may prevent agents from breaking ties around symmetric conflict (e.g., bottlenecks) and might lead to live-/deadlocks. In this paper, we propose SYLPH, a novel learning-based MAPF framework aimed to mitigate the adverse effects of homogeneity by allowing agents to learn and dynamically select different social behaviors (akin to individual, dynamic roles), without affecting the scalability offered by parameter sharing. Specifically, SYLPH offers a novel hierarchical mechanism by introducing Social Value Orientation (SVO) as a temporally extended latent variable that plays a central role in both policy generation and reward assignment. To support this hierarchical decision-making process, we introduce Social-aware Multi-Policy PPO (SMP3O), a reinforcement learning method that ensures stable and effective training through a mechanism for the cross-utilization of advantages. Moreover, we design an SVO-based learning tie-breaking algorithm, allowing agents to proactively avoid collisions, rather than relying solely on post-processing techniques. As a result of this hierarchical decision-making and exchange of social preferences, SYLPH endows agents with the ability to reason about the MAPF task through more latent spaces and nuanced contexts, leading to varied responses that can help break ties around symmetric conflicts. Our comparative experiments show that SYLPH achieves state-of-the-art performance, surpassing other learning-based MAPF planners in random, room-like, and maze-like maps, while our ablation studies demonstrate the advantages of each component in SYLPH. We finally experimentally validate our trained policies on hardware in three types of maps, showing how SYLPH allows agents to find high-quality paths under real-life conditions. Our code and videos are available at: marmotlab.github.io/mapf_sylph.

PDF Details DOI

ICRA Conference 2025 Conference Paper

DARE: Diffusion Policy for Autonomous Robot Exploration

  • Yuhong Cao
  • Jeric Lew
  • Jingsong Liang
  • Jin Cheng 0002
  • Guillaume Sartoretti

Autonomous robot exploration requires a robot to efficiently explore and map unknown environments. Compared to conventional methods that can only optimize paths based on the current robot belief, learning-based methods show the potential to achieve improved performance by drawing on past experiences to reason about unknown areas. In this paper, we propose DARE, a novel generative approach that leverages diffusion models trained on expert demonstrations, which can explicitly generate an exploration path through one-time inference. We build DARE upon an attention-based encoder and a diffusion model, and introduce ground truth optimal demonstrations for training to learn better patterns for exploration. The trained planner can reason about the partial belief to recognize the potential structure in unknown areas and consider these areas during path planning. Our experiments demonstrate that DARE achieves on-par performance with both conventional and learning-based state-of-the-art exploration planners, as well as good generalizability in both simulations and real-life scenarios. Our code is available at github.com/marmotlab/DARE.

Details

ICRA Conference 2025 Conference Paper

Deploying Ten Thousand Robots: Scalable Imitation Learning for Lifelong Multi-Agent Path Finding

  • He Jiang
  • Yutong Wang 0003
  • Rishi Veerapaneni
  • Tanishq Duhan
  • Guillaume Sartoretti
  • Jiaoyang Li 0001

Lifelong Multi-Agent Path Finding (LMAPF) repeatedly finds collision-free paths for multiple agents that are continually assigned new goals when they reach current ones. Recently, this field has embraced learning-based methods, which reactively generate single-step actions based on individual local observations. However, it is still challenging for them to match the performance of the best search-based algorithms, especially in large-scale settings. This work proposes an imitation-learning-based LMAPF solver that introduces a novel communication module as well as systematic single-step collision resolution and global guidance techniques. Our proposed solver, Scalable Imitation Learning for LMAPF (SILLM), inherits the fast reasoning speed of learning-based methods and the high solution quality of search-based methods with the help of modern GPUs. Across six large-scale maps with up to $\mathbf{1 0, 0 0 0}$ agents and varying obstacle structures, SILLM surpasses the best learning- and search-based baselines, achieving average throughput improvements of $\mathbf{137. 7 \%}$ and $\mathbf{1 6. 0 \%}$, respectively. Furthermore, SILLM also beats the winning solution of the 2023 League of Robot Runners, an international LMAPF competition. Finally, we validated SILLM with 10 real robots and $\mathbf{1 0 0}$ virtual robots in a mock warehouse environment.

Details

AAAI Conference 2025 Conference Paper

LNS2+RL: Combining Multi-agent Reinforcement Learning with Large Neighborhood Search in Multi-agent Path Finding

  • Yutong Wang
  • Tanishq Duhan
  • Jiaoyang Li
  • Guillaume Sartoretti

Multi-Agent Path Finding (MAPF) is a critical component of logistics and warehouse management, which focuses on planning collision-free paths for a team of robots in a known environment. Recent work introduced a novel MAPF approach, LNS2, which proposed to repair a quickly-obtainable set of infeasible paths via iterative re-planning, by relying on a fast, yet lower-quality, priority-based planner. At the same time, there has been a recent push for Multi-Agent Reinforcement Learning (MARL) based MAPF algorithms, which let agents learn decentralized policies that exhibit improved cooperation over such priority planning, although inevitably remaining slower. In this paper, we introduce a new MAPF algorithm, LNS2+RL, which combines the distinct yet complementary characteristics of LNS2 and MARL to effectively balance their individual limitations and get the best from both worlds. During early iterations, LNS2+RL relies on MARL for low-level re-planning, which we show eliminates collisions much more than a priority-based planner. There, our MARL-based planner allows agents to reason about past and future/predicted information to gradually learn cooperative decision-making through a finely designed curriculum learning. At later stages of planning, LNS2+RL adaptively switches to priority-based planning to quickly resolve the remaining collisions, naturally trading-off solution quality and computational efficiency. Our comprehensive experiments on challenging tasks across various team sizes, world sizes, and map structures consistently demonstrate the superior performance of LNS2+RL compared to many MAPF algorithms, including LNS2, LaCAM, and EECBS. In maps with complex structures, the advantages of LNS2+RL are particularly pronounced, with LNS2+RL achieving a success rate of over 50% in nearly half of the tested tasks, while that of LaCAM and EECBS falls to 0%.

PDF Details DOI

ICRA Conference 2025 Conference Paper

MARVEL: Multi-Agent Reinforcement Learning for Constrained Field-of-View Multi-Robot Exploration in Large-Scale Environments

  • Jimmy Chiun
  • Shizhe Zhang
  • Yizhuo Wang 0004
  • Yuhong Cao
  • Guillaume Sartoretti

In multi-robot exploration, a team of mobile robot is tasked with efficiently mapping an unknown environments. While most exploration planners assume omnidirectional sensors like LiDAR, this is impractical for small robots such as drones, where lightweight, directional sensors like cameras may be the only option due to payload constraints. These sensors have a constrained field-of-view (FoV), which adds complexity to the exploration problem, requiring not only optimal robot positioning but also sensor orientation during movement. In this work, we propose MARVEL, a neural framework that leverages graph attention networks, together with novel frontiers and orientation features fusion technique, to develop a collaborative, decentralized policy using multi-agent reinforcement learning (MARL) for robots with constrained FoV. To handle the large action space of viewpoints planning, we further introduce a novel information-driven action pruning strategy. MARVEL improves multi-robot coordination and decision-making in challenging large-scale indoor environments, while adapting to various team sizes and sensor configurations (i. e. , FoV and sensor range) without additional training. Our extensive evaluation shows that MARVEL's learned policies exhibit effective coordinated behaviors, outperforming state-of-the-art exploration planners across multiple metrics. We experimentally demonstrate MARVEL's generalizability in large-scale environments, of up to 90 m by 90 m, and validate its practical applicability through successful deployment on a team of real drone hardware.

Details

NeurIPS Conference 2025 Conference Paper

Preference-Driven Multi-Objective Combinatorial Optimization with Conditional Computation

  • Mingfeng Fan
  • Jianan Zhou
  • Yifeng Zhang
  • Yaoxin Wu
  • Jinbiao Chen
  • Guillaume Sartoretti

Recent deep reinforcement learning methods have achieved remarkable success in solving multi-objective combinatorial optimization problems (MOCOPs) by decomposing them into multiple subproblems, each associated with a specific weight vector. However, these methods typically treat all subproblems equally and solve them using a single model, hindering the effective exploration of the solution space and thus leading to suboptimal performance. To overcome the limitation, we propose POCCO, a novel plug-and-play framework that enables adaptive selection of model structures for subproblems, which are subsequently optimized based on preference signals rather than explicit reward values. Specifically, we design a conditional computation block that routes subproblems to specialized neural architectures. Moreover, we propose a preference-driven optimization algorithm that learns pairwise preferences between winning and losing solutions. We evaluate the efficacy and versatility of POCCO by applying it to two state-of-the-art neural methods for MOCOPs. Experimental results across four classic MOCOP benchmarks demonstrate its significant superiority and strong generalization.

PDF Details

ICRA Conference 2025 Conference Paper

SIGMA: Sheaf-Informed Geometric Multi-Agent Pathfinding

  • Shuhao Liao
  • Weihang Xia
  • Yuhong Cao
  • Weiheng Dai
  • Chengyang He
  • Wenjun Wu 0001
  • Guillaume Sartoretti

The Multi-Agent Path Finding (MAPF) problem aims to determine the shortest and collision-free paths for multiple agents in a known, potentially obstacle-ridden environment. It is the core challenge for robotic deployments in large-scale logistics and transportation. Decentralized learningbased approaches have shown great potential for addressing the MAPF problems, offering more reactive and scalable solutions. However, existing learning-based MAPF methods usually rely on agents making decisions based on a limited field of view (FOV), resulting in short-sighted policies and inefficient cooperation in complex scenarios. There, a critical challenge is to achieve consensus on potential movements between agents based on limited observations and communications. To tackle this challenge, we introduce a new framework that applies sheaf theory to decentralized deep reinforcement learning, enabling agents to learn geometric cross-dependencies between each other through local consensus and utilize them for tightly cooperative decision-making. In particular, sheaf theory provides a mathematical proof of conditions for achieving global consensus through local observation. Inspired by this, we incorporate a neural network to approximately model the consensus in latent space based on sheaf theory and train it through self-supervised learning. During the task, in addition to normal features for MAPF as in previous works, each agent distributedly reasons about a learned consensus feature, leading to efficient cooperation on pathfinding and collision avoidance. As a result, our proposed method demonstrates significant improvements over state-of-the-art learning-based MAPF planners, especially in relatively large and complex scenarios, demonstrating its superiority over baselines in various simulations and real-world robot experiments.

Details

AIJ Journal 2025 Journal Article

Social behavior as a key to learning-based multi-agent pathfinding dilemmas

  • Chengyang He
  • Tanishq Duhan
  • Parth Tulsyan
  • Patrick Kim
  • Guillaume Sartoretti

The Multi-agent Path Finding (MAPF) problem involves finding collision-free paths for a team of agents in a known, static environment, with important applications in warehouse automation, logistics, or last-mile delivery. To meet the needs of these large-scale applications, current learning-based methods often deploy the same fully trained, decentralized network to all agents to improve scalability. However, such parameter sharing typically results in homogeneous behaviors among agents, which may prevent agents from breaking ties around symmetric conflict (e. g. , bottlenecks) and might lead to live-/deadlocks. In this paper, we propose SYLPH, a novel learning-based MAPF framework aimed to mitigate the adverse effects of homogeneity by allowing agents to learn and dynamically select different social behaviors (akin to individual, dynamic roles), without affecting the scalability offered by parameter sharing. Specifically, SYLPH agents learn to select their Social Value Orientation (SVO) given the situation at hand, quantifying their own level of selfishness/altruism, as well as an SVO-conditioned MAPF policy dictating their movement actions. To these ends, each agent first determines the most influential other agent in the system by predicting future conflicts/interactions with other agents. Each agent selects its own SVO towards that agent, and trains its decentralized MAPF policy to enact this SVO until another agent becomes more influential. To further allow agents to consider each others' social preferences, each agent gets access to the SVO value of their neighbors. As a result of this hierarchical decision-making and exchange of social preferences, SYLPH endows agents with the ability to reason about the MAPF task through more latent spaces and nuanced contexts, leading to varied responses that can help break ties around symmetric conflicts. [. .. ]

Details DOI

ICRA Conference 2024 Conference Paper

ALPHA: Attention-based Long-horizon Pathfinding in Highly-structured Areas

  • Chengyang He
  • Tianze Yang
  • Tanishq Duhan
  • Yutong Wang 0003
  • Guillaume Sartoretti

The multi-agent pathfinding (MAPF) problem seeks collision-free paths for a team of agents from their current positions to their pre-set goals in a known environment, and is an essential problem found at the core of many logistics, transportation, and general robotics applications. Existing learning-based MAPF approaches typically only let each agent make decisions based on a limited field-of-view (FOV) around its position, as a natural means to fix the input dimensions of its policy network. However, this often makes policies shortsighted, since agents lack the ability to perceive and plan for obstacles/agents beyond their FOV. To address this challenge, we propose ALPHA, a new framework combining the use of ground truth proximal (local) information and fuzzy distal (global) information to let agents sequence local decisions based on the full current state of the system, and avoid such myopicity. We further allow agents to make short-term predictions about each others’ paths, as a means to reason about each others’ path intentions, thereby enhancing the level of cooperation among agents at the whole system level. Our neural structure relies on a Graph Transformer architecture to allow agents to selectively combine these different sources of information and reason about their inter-dependencies at different spatial scales. Our simulation experiments demonstrate that ALPHA outperforms both globally-guided MAPF solvers and communication-learning based ones, showcasing its potential towards scalability in realistic deployments.

Details

AAMAS Conference 2024 Conference Paper

Attention-based Priority Learning for Limited Time Multi-Agent Path Finding

  • Yibin Yang
  • Mingfeng Fan
  • Chengyang He
  • Jianqiang Wang
  • Heye Huang
  • Guillaume Sartoretti

Solving large-scale Multi-Agent Path Finding (MAPF) within a limited time remains an open challenge, despite its importance for many robotic applications. Recent learning-based methods scale better than conventional ones, but remain suboptimal and often exhibit low success rates within a limited time on large-scale instances. These limitations often stem from their black-box nature. In this study, we propose a hybrid approach that incorporates prioritized planning with learning-based methods to explicitly address these challenges. We formulate prioritized planning as a Markov Decision Process and introduce a reinforcement learningbased prioritized planning paradigm. In doing so, we develop a novel Synthetic Score-based Attention Network (S2AN) to learn conflict/blocking relationships among agents, and deliver blocking-free priorities. By integrating priority mechanisms and leveraging a new attention-based neural network for enhanced multi-agent cooperative strategies, our method enhances solution completeness while trading off scalability and maintains linear time complexity, thus offering a robust avenue for large-scale MAPF tasks. Comparisons demonstrate its superiority over current learning-based methods in terms of solution quality, completeness, and reachability within limited time constraints, especially in largescale scenarios. Moreover, an extensive set of numerical results reveals superior completeness compared to restricted-time Priority- Based Search (PBS) and Priority Inheritance with Backtracking (PIBT) in medium to large-scale obstacle-dense scenarios.

PDF

AAMAS Conference 2024 Conference Paper

Collaborative Deep Reinforcement Learning for Solving Multi-Objective Vehicle Routing Problems

  • Yaoxin Wu
  • Mingfeng Fan
  • Zhiguang Cao
  • Ruobin Gao
  • Yaqing Hou
  • Guillaume Sartoretti

Existing deep reinforcement learning (DRL) methods for multiobjective vehicle routing problems (MOVRPs) typically decompose an MOVRP into subproblems with respective preferences and then train policies to solve corresponding subproblems. However, such a paradigm is still less effective in tackling the intricate interactions among subproblems, thus holding back the quality of the Pareto solutions. To counteract this limitation, we introduce a collaborative deep reinforcement learning method. We first propose a preferencebased attention network (PAN) that allows the DRL agents to reason out solutions to subproblems in parallel, where a shared encoder learns the instance embedding and a decoder is tailored for each agent by preference intervention to construct respective solutions. Then, we design a collaborative active search (CAS) to further improve the solution quality, which updates only a part of the decoder parameters per instance during inference. In the CAS process, we also explicitly foster the interactions of neighboring DRL agents by imitation learning, empowering them to exchange insights of elite solutions to similar subproblems. Extensive results on random and benchmark instances verified the efficacy of PAN and CAS, which is particularly pronounced on the configurations (i. e. , problem sizes or node distributions) beyond the training ones. Our code is available at https: //github. com/marmotlab/PAN-CAS.

PDF

IROS Conference 2024 Conference Paper

DecAP: Decaying Action Priors for Accelerated Imitation Learning of Torque-Based Legged Locomotion Policies

  • Shivam Sood
  • Ge Sun
  • Peizhuo Li
  • Guillaume Sartoretti

Optimal Control for legged robots has gone through a paradigm shift from position-based to torque-based control, owing to the latter’s compliant and robust nature. In parallel to this shift, the community has also turned to Deep Reinforcement Learning (DRL) as a promising approach to directly learn locomotion policies for complex real-life tasks. However, most end-to-end DRL approaches still operate in position space, mainly because learning in torque space is often sample-inefficient and does not consistently converge to natural gaits. To address these challenges, we propose a two-stage framework. In the first stage, we generate our own imitation data by training a position-based policy, eliminating the need for expert knowledge to design optimal controllers. The second stage incorporates decaying action priors, a novel method to enhance the exploration of torque-based policies aided by imitation rewards. We show that our approach consistently outperforms imitation learning alone and is robust to scaling these rewards from 0. 1x to 10x. We further validate the benefits of torque control by comparing the robustness of a position-based policy to a position-assisted torque-based policy on a quadruped (Unitree Go1) without any domain randomization in the form of external disturbances during training. 3

Details

ICRA Conference 2024 Conference Paper

Dynamic Coalition Formation and Routing for Multirobot Task Allocation via Reinforcement Learning

  • Weiheng Dai
  • Aditya Bidwai
  • Guillaume Sartoretti

Many multi-robot deployments, such as automated construction of buildings, distributed search, or cooperative mapping, often require agents to intelligently coordinate their trajectories and form coalition over a large domain, to complete spatially distributed tasks as quickly as possible. We focus on scenarios involving homogeneous robots, but where tasks vary in the number of agents required to start them. For example, construction robots may need to collaboratively air-lift heavy objects at different locations (e. g. , prefabricated rooms, crates of material/equipment), where the weight of each payload defines the required coalition size. To balance the total travel time of the agents and their waiting time (before task initiation), agents need to carefully sequence tasks but also dynamically form/disband coalitions. While simpler problems can be approached using heuristics or optimization, these methods struggle with more complex instances involving large task-to-agent ratios, where frequent coalition changes are needed. In this work, we propose to let agents learn to iteratively build cooperative schedules to solve such problems, by casting the problem in the reinforcement learning framework. Our approach relies on an attention-based neural network, allowing agents to reason about the current state of the system to sequence movement decisions that optimize short-term coalition formation and long-term task scheduling. We further propose a novel leader-follower technique to boost cooperation learning and compare our performance to conventional baselines in a wide variety of scenarios. There, our method closely matches or outperforms the baselines; in particular, it yields higher-quality solutions and is at least 2 orders of magnitude faster than exact solver in cases where frequent coalition updates are required.

Details

IROS Conference 2024 Conference Paper

HeteroLight: A General and Efficient Learning Approach for Heterogeneous Traffic Signal Control

  • Yifeng Zhang
  • Peizhuo Li
  • Mingfeng Fan
  • Guillaume Sartoretti

Efficient and scalable adaptive traffic signal control is crucial in reducing congestion, maximizing through-put, and improving mobility experience in ever-expanding cities. Recent advances in multi-agent reinforcement learning (MARL) with parameter sharing have significantly improved the adaptive optimization of large-scale, complex, and dynamic traffic flows. However, the limited model representation capability due to shared parameters impedes the learning of diverse control strategies for intersections with different flows/topologies, posing significant challenges to achieving effective signal control in complex and varied real-world traffic scenarios. To address these challenges, we present a novel MARL-based general traffic signal control framework, called HeteroLight. Specifically, we first introduce a General Feature Extraction (GFE) module, crafted in a decoder-only fashion, where we employ an attention mechanism to facilitate efficient and flexible extraction of traffic dynamics at intersections with varied topologies. Additionally, we incorporate an Intersection Specifics Extraction (ISE) module, designed to identify key latent vectors that represent the unique intersection’s topology and traffic dynamics through variational inference techniques. By integrating the learned intersection-specific information into policy learning, we enhance the parameter-sharing mechanism, improving the model’s representation diversity among different agents and enabling the learning of a more efficient shared control strategy. Through comprehensive evaluations against other state-of-the-art traffic signal control methods on the real-world Monaco traffic network, our empirical findings reveal that HeteroLight consistently outperforms other methods across various evaluation metrics, highlighting its superiority in optimizing traffic flows in heterogeneous traffic networks.

Details

IROS Conference 2024 Conference Paper

IR 2: Implicit Rendezvous for Robotic Exploration Teams under Sparse Intermittent Connectivity

  • Derek Ming Siang Tan
  • Yixiao Ma
  • Jingsong Liang
  • Yi Cheng Chng
  • Yuhong Cao
  • Guillaume Sartoretti

Information sharing is critical in time-sensitive and realistic multi-robot exploration, especially for smaller robotic teams in large-scale environments where connectivity may be sparse and intermittent. Existing methods often overlook such communication constraints by assuming unrealistic global connectivity. Other works account for communication constraints (by maintaining close proximity or line of sight during information exchange), but are often inefficient. For instance, preplanned rendezvous approaches typically involve unnecessary detours resulting from poorly timed rendezvous, while pursuit-based approaches often result in short-sighted decisions due to their greedy nature. We present IR 2, a deep reinforcement learning approach to information sharing for multi-robot exploration. Leveraging attention-based neural networks trained via reinforcement and curriculum learning, IR 2 allows robots to effectively reason about the longer-term trade-offs between disconnecting for solo exploration and reconnecting for information sharing. In addition, we propose a hierarchical graph formulation to maintain a sparse yet informative graph, enabling our approach to scale to large-scale environments. We present simulation results in three large-scale Gazebo environments, which show that our approach yields 6. 6−34. 1% shorter exploration paths and significantly improved mapped area consistency among robots when compared to state-of-the-art baselines. Our simulation training and testing code is available at https://github.com/marmotlab/IR2.

Details

ICRA Conference 2024 Conference Paper

Learning Heterogeneous Multi-Agent Allocations for Ergodic Search

  • Ananya Rao
  • Guillaume Sartoretti
  • Howie Choset

Information-based coverage directs robots to move over an area to optimize a pre-defined objective function based on some measure of information. Our prior work determined that the spectral decomposition of an information map can be used to guide a set of heterogeneous agents, each with different sensor and motion models, to optimize coverage in a target region, based on a measure called ergodicity. In this paper, we build on this insight to construct a reinforcement learning formulation of the problem of allocating heterogeneous agents to different search regions in the frequency domain. We relate the spectral coefficients of the search map to each other in three different ways. The first method maps agents to predefined sets of spectral coefficients. In the second method, each agent learns a weight distribution over all spectral coefficients. Finally, in the third method, each agent learns weight distributions as parameterized curves over coefficients. Our numerical results demonstrate that distributing and assigning coverage responsibilities to agents depending on their sensing and motion models leads to 40%, 51%, and 46% improvement in coverage performance as measured by the ergodic metric, and 15%, 22%, and 20% improvement in time to find all targets in the search region, for the three methods respectively.

Details

IROS Conference 2024 Conference Paper

Learning-based Hierarchical Control: Emulating the Central Nervous System for Bio-Inspired Legged Robot Locomotion

  • Ge Sun
  • Milad Shafiee
  • Peizhuo Li
  • Guillaume Bellegarda
  • Auke Jan Ijspeert
  • Guillaume Sartoretti

Animals possess a remarkable ability to navigate challenging terrains, achieved through the interplay of various pathways between the brain, central pattern generators (CPGs) in the spinal cord, and musculoskeletal system. Traditional bioinspired control frameworks often rely on a singular control policy that models both higher (supraspinal) and spinal cord functions. In this work, we build upon our previous research by introducing two distinct neural networks: one tasked with modulating the frequency and amplitude of CPGs to generate the basic locomotor rhythm (referred to as the spinal policy), and the other responsible for receiving environmental perception data and directly modulating the rhythmic output from the spinal policy to execute precise movements on challenging terrains (referred to as the descending modulation policy). This division of labor more closely mimics the hierarchical locomotor control systems observed in legged animals, thereby enhancing the robot’s ability to navigate various uneven surfaces, including steps, high obstacles, and terrains with gaps. Additionally, we investigate the impact of sensorimotor delays within our framework, validating several biological assumptions about animal locomotion systems. Specifically, we demonstrate that spinal circuits play a crucial role in generating the basic locomotor rhythm, while descending pathways are essential for enabling appropriate gait modifications to accommodate uneven terrain. Notably, our findings also reveal that the multi-layered control inherent in animals exhibits remarkable robustness against sensorimotor delays. These findings advance our understanding of the fundamental principles governing the interplay between spinal and supraspinal mechanisms in biological locomotion. Moreover, they inform the design of bioinspired locomotion controllers that emulate these biological structures, facilitating natural movement in complex and realistic environments.

Details

IROS Conference 2024 Conference Paper

Reconfigurable Multi-Rotor for High-Precision Physical Interaction

  • Joshua Taylor
  • Nursultan Imanberdiyev
  • Meng Yee Michael Chuah
  • Wei-Yun Yau
  • Guillaume Sartoretti
  • Efe Camci

Unmanned aerial vehicles (UAVs) for contact-based tasks at height can greatly improve the safety of the human workers involved. However, performing contact-based tasks with typical under-actuated UAVs is non-trivial. Due to their coupled translational and rotational dynamics and their limited station-keeping performance under physical disturbances, it is difficult to maintain precise and consistent contact. We address these problems in the context of physical interaction with vertical, cylindrical target objects, such as trees. We present a novel UAV design with a pair of tilt-rotors and a landing gear that can reconfigure into a front-mounted, two-fingered gripper. While the tilt-rotors provide horizontal force toward the target object without pitching the UAV forward, the reconfigurable landing gear enables the UAV to obtain support from the target object. Such support results in an approximately 80% improvement in position- and heading-keeping performance. Moreover, the landing gear is designed as a cable-driven under-actuated system, which requires only one actuator to control both the reconfiguration and the grasping (i. e. , five degrees of freedom in total). Such a minimalist design helps keep the UAV power consumption for interactions low. This marks progress towards safe, high-precision physical interaction against vertical, cylindrical target objects. Our UAV in action: https://youtu.be/D-65vldox_A.

Details

ICRA Conference 2023 Conference Paper

ARiADNE: A Reinforcement learning approach using Attention-based Deep Networks for Exploration

  • Yuhong Cao
  • Tianxiang Hou
  • Yizhuo Wang 0004
  • Xian Yi
  • Guillaume Sartoretti

In autonomous robot exploration tasks, a mobile robot needs to actively explore and map an unknown environment as fast as possible. Since the environment is being revealed during exploration, the robot needs to frequently re-plan its path online, as new information is acquired by onboard sensors and used to update its partial map. While state-of-the-art exploration planners are frontier- and sampling-based, encouraged by the recent development in deep reinforcement learning (DRL), we propose ARiADNE, an attention-based neural approach to obtain real-time, non-myopic path planning for autonomous exploration. ARiADNE is able to learn dependencies at multiple spatial scales between areas of the agent's partial map, and implicitly predict potential gains associated with exploring those areas. This allows the agent to sequence movement actions that balance the natural trade-off between exploitation/refinement of the map in known areas and exploration of new areas. We experimentally demonstrate that our method outperforms both learning and non-learning state-of-the-art baselines in terms of average trajectory length to complete exploration in hundreds of simplified 2D indoor scenarios. We further validate our approach in high-fidelity Robot Operating System (ROS) simulations, where we consider a real sensor model and a realistic low-level motion controller, toward deployment on real robots.

Details

JAAMAS Journal 2023 Journal Article

Full communication memory networks for team-level cooperation learning

  • Yutong Wang
  • Yizhuo Wang
  • Guillaume Sartoretti

Abstract Communication in multi-agent systems is a key driver of team-level cooperation, for instance allowing individual agents to augment their knowledge about the world in partially-observable environments. In this paper, we propose two reinforcement learning-based multi-agent models, namely FCMNet and FCMTran. The two models both allow agents to simultaneously learn a differentiable communication mechanism that connects all agents as well as a common, cooperative policy conditioned upon received information. FCMNet utilizes multiple directional Long Short-Term Memory chains to sequentially transmit and encode the current observation-based messages sent by every other agent at each timestep. FCMTran further relies on the encoder of a modified transformer to simultaneously aggregate multiple self-generated messages sent by all agents at the previous timestep into a single message that is used in the current timestep. Results from evaluating our models on a challenging set of StarCraft II micromanagement tasks with shared rewards show that FCMNet and FCMTran both outperform recent communication-based methods and value decomposition methods in almost all tested StarCraft II micromanagement tasks. We further improve the performance of our models by combining them with value decomposition techniques; there, in particular, we show that FCMTran with value decomposition significantly pushes the state-of-the-art on one of the hardest benchmark tasks without any task-specific tuning. We also investigate the robustness of FCMNet under communication disturbances (i. e. , binarized messages, random message loss, and random communication order) in an asymmetric collaborative pathfinding task with individual rewards, demonstrating FMCNet’s potential applicability in real-world robotic tasks.

Details DOI

ICML Conference 2023 Conference Paper

Learning Temporally AbstractWorld Models without Online Experimentation

  • Benjamin Freed
  • Siddarth Venkatraman
  • Guillaume Sartoretti
  • Jeff G. Schneider
  • Howie Choset

Agents that can build temporally abstract representations of their environment are better able to understand their world and make plans on extended time scales, with limited computational power and modeling capacity. However, existing methods for automatically learning temporally abstract world models usually require millions of online environmental interactions and incentivize agents to reach every accessible environmental state, which is infeasible for most real-world robots both in terms of data efficiency and hardware safety. In this paper, we present an approach for simultaneously learning sets of skills and temporally abstract, skill-conditioned world models purely from offline data, enabling agents to perform zero-shot online planning of skill sequences for new tasks. We show that our approach performs comparably to or better than a wide array of state-of-the-art offline RL algorithms on a number of simulated robotics locomotion and manipulation benchmarks, while offering a higher degree of adaptability to new goals. Finally, we show that our approach offers a much higher degree of robustness to perturbations in environmental dynamics, compared to policy-based methods.

Details

IROS Conference 2023 Conference Paper

SCRIMP: Scalable Communication for Reinforcement- and Imitation-Learning-Based Multi-Agent Pathfinding

  • Yutong Wang 0003
  • Bairan Xiang
  • Shinan Huang
  • Guillaume Sartoretti

Trading off performance guarantees in favor of scalability, the Multi-Agent Path Finding (MAPF) community has recently started to embrace Multi-Agent Reinforcement Learning (MARL), where agents learn to collaboratively generate individual, collision-free (but often suboptimal) paths. Scalability is usually achieved by assuming a local field of view (FOV) around the agents, helping scale to arbitrary world sizes. However, this assumption significantly limits the amount of information available to the agents, making it difficult for them to enact the type of joint maneuvers needed in denser MAPF tasks. In this paper, we propose SCRIMP, where agents learn individual policies from even very small (down to $3\mathrm{x}3$ ) FOVs, by relying on a highly-scalable global communication mechanism based on a modified transformer. We further equip agents with a state-value-based tie-breaking strategy to further improve performance in symmetric situations, and introduce intrinsic rewards to encourage exploration while mitigating the long-term credit assignment problem. Empirical evaluations on a set of experiments indicate that SCRIMP can achieve higher performance with improved scalability compared to other state-of-the-art learning-based MAPF planners with larger FOVs, and even yields similar performance as a classical centralized planner in many cases. Ablation studies further validate the effectiveness of our proposed techniques. Finally, we show that our trained model can be directly implemented on real robots for online MAPF through high-fidelity simulations in gazebo.

Details

AAMAS Conference 2023 Conference Paper

SCRIMP: Scalable Communication for Reinforcement- and Imitation-Learning-Based Multi-Agent Pathfinding

  • Yutong Wang
  • Bairan Xiang
  • Shinan Huang
  • Guillaume Sartoretti

In this paper, we propose SCRIMP, a multi-agent reinforcement learning approach for multi-agent path finding. Our method learns individual policies from very small FOVs (3x3), by relying on a highly-scalable global/local communication mechanism based on a modified transformer. We further introduce a state-value-based tiebreaking strategy to improve performance in symmetric situations and intrinsic rewards to encourage exploration while mitigating the long-term credit assignment problem. Empirical evaluations indicate that SCRIMP can outperform other state-of-the-art learningbased planners with larger FOVs and even yield similar performance as a classical centralized planner.

PDF

AAMAS Conference 2023 Conference Paper

SocialLight: Distributed Cooperation Learning towards Network-Wide Traffic Signal Control

  • Harsh Goel
  • Yifeng Zhang
  • Mehul Damani
  • Guillaume Sartoretti

Many recent works have turned to multi-agent reinforcement learning (MARL) for adaptive traffic signal control to optimize the travel time of vehicles over large urban networks. However, achieving effective and scalable cooperation among junctions (agents) remains an open challenge, as existing methods often rely on extensive, nongeneralizable reward shaping or on non-scalable centralized learning. To address these problems, we propose a new MARL method for traffic signal control, SocialLight, which learns cooperative traffic control policies by distributedly estimating the individual marginal contribution of agents on their local neighborhood. SocialLight relies on the Asynchronous Actor Critic (A3C) framework, and makes learning scalable by learning a locally-centralized critic conditioned over the states and actions of neighboring agents, used by agents to estimate individual contributions by counterfactual reasoning. We further introduce important modifications to the advantage calculation that help stabilize policy updates. These modifications decouple the impact of the neighbors’ actions on the computed advantages, thereby reducing the variance in the gradient updates. We benchmark our trained network against state-of-the-art traffic signal control methods on standard benchmarks in two traffic simulators, SUMO and CityFlow. Our results show that SocialLight exhibits improved scalability to larger road networks and better performance across usual traffic metrics.

PDF

IROS Conference 2023 Conference Paper

Spatio-Temporal Attention Network for Persistent Monitoring of Multiple Mobile Targets

  • Yizhuo Wang 0004
  • Yutong Wang 0003
  • Yuhong Cao
  • Guillaume Sartoretti

This work focuses on the persistent monitoring problem, where a set of targets moving based on an unknown model must be monitored by an autonomous mobile robot with a limited sensing range. To keep each target's position estimate as accurate as possible, the robot needs to adaptively plan its path to (re-)visit all the targets and update its belief from measurements collected along the way. In doing so, the main challenge is to strike a balance between exploitation, i. e. , re-visiting previously-located targets, and exploration, i. e. , finding new targets or re-acquiring lost ones. Encouraged by recent advances in deep reinforcement learning, we introduce an attention-based neural solution to the persistent monitoring problem, where the agent can learn the inter-dependencies between targets, i. e. , their spatial and temporal correlations, conditioned on past measurements. This endows the agent with the ability to determine which target, time, and location to attend to across multiple scales, which we show also helps relax the usual limitations of a finite target set with prior positional information. We experimentally demonstrate that our method outperforms other baselines in terms of number of targets visits and average estimation error in complex environments. Finally, we implement and validate our model in a drone-based simulation experiment to monitor mobile ground targets in a high-fidelity simulator.

Details

AAMAS Conference 2022 Conference Paper

FCMNet: Full Communication Memory Net for Team-Level Cooperation in Multi-Agent Systems

  • Yutong Wang
  • Guillaume Sartoretti

Decentralized cooperation in partially-observable multi-agent systems requires effective communications among agents. To support this effort, this work focuses on the class of problems where global communications are available but may be unreliable, thus precluding differentiable communication learning methods. We introduce FCMNet, a reinforcement learning based approach that allows agents to simultaneously learn a) an effective multi-hop communications protocol and b) a common, decentralized policy that enables team-level decision-making. Specifically, our proposed method utilizes the hidden states of multiple directional recurrent neural networks as communication messages among agents. Using a simple multi-hop topology, we endow each agent with the ability to receive information sequentially encoded by every other agent at each time step, leading to improved global cooperation. We demonstrate FCMNet on a challenging set of StarCraft II micromanagement tasks with shared rewards, as well as a collaborative multi-agent pathfinding task with individual rewards. There, our comparison results show that FCMNet outperforms state-of-the-art communication-based reinforcement learning methods in all Star- Craft II micromanagement tasks, and value decomposition methods in certain tasks. We further investigate the robustness of FCM- Net under realistic communication disturbances, such as random message loss or binarized messages (i. e. , non-differentiable communication channels), to showcase FMCNet’s potential applicability to robotic tasks under a variety of real-world conditions.

PDF

ICRA Conference 2022 Conference Paper

Multi-Agent Dynamic Ergodic Search with Low-Information Sensors

  • Howard Coffin
  • Ian Abraham
  • Guillaume Sartoretti
  • Tyler Dillstrom
  • Howie Choset

The long-term goal of this work is to enable agents with low-information sensors to perform tasks usually restricted to ones with more sophisticated, high-information sensing capabilities. Our approach is to regulate the motion of these low-information agents to obtain “high-information” results. As a first step, we consider a multi-agent system tasked with locating and tracking a moving target using only noisy binary sensors that measure the presence (or lack thereof) of a target in the sensor's field of view. To generate effective paths for these agents, we use ergodic trajectory optimization with a novel mutual information map that is fast to compute and can handle the discontinuous measurement models often associated with low-information sensing. We compare our approach with existing motion planning methods in multiple simulated experiments. Our experiments show that agents using our method outperform purely coverage-based approaches as well as naive ergodic approaches.

Details

ICRA Conference 2021 Conference Paper

Autonomous Decentralized Shape-Based Navigation for Snake Robots in Dense Environments

  • Guillaume Sartoretti
  • Tianyu Wang 0010
  • Gabriel Chuang
  • Qingyang Li
  • Howie Choset

In this work, we focus on the autonomous navigation of snake robots in densely-cluttered environments, where collisions between the robot and obstacles are frequent, which could happen often in disaster scenarios, underground caves, or grassland/forest environments. This work takes the view that obstacles are not to be avoided, but rather exploited to support and direct the motion of the snake robot. We build upon a decentralized state-of-the-art compliant controller for serpenoid locomotion, and develop a bi-stable dynamical system that relies on inertial feedback to continuously steer the robot toward a desired direction. We experimentally show that this controller allows the robot to autonomously navigate dense environments by consistently locomoting along a given, global direction of travel in the world, which could be selected by a human operator or a higher level planner. We further equip the robot with an onboard vision system, allowing the robot to autonomously select its own direction of travel, based on the obstacle distribution ahead of its position (i. e. , enacting feedforward control). In those additional experiments on hardware, we show how such an exteroceptive sensor can allow the robot to steer before hitting obstacles and to preemptively avoid challenging regions where proprioception-only (i. e. , torque and inertial) feedback control would not suffice.

Details

AAAI Conference 2020 Conference Paper

Communication Learning via Backpropagation in Discrete Channels with Unknown Noise

  • Benjamin Freed
  • Guillaume Sartoretti
  • Jiaheng Hu
  • Howie Choset

This work focuses on multi-agent reinforcement learning (RL) with inter-agent communication, in which communication is differentiable and optimized through backpropagation. Such differentiable approaches tend to converge more quickly to higher-quality policies compared to techniques that treat communication as actions in a traditional RL framework. However, modern communication networks (e. g. , Wi- Fi or Bluetooth) rely on discrete communication channels, for which existing differentiable approaches that consider realvalued messages cannot be directly applied, or require biased gradient estimators. Some works have overcome this problem by treating the message space as an extension of the action space, and use standard RL to optimize message selection, but these methods tend to converge slower and to inferior policies. In this paper, we propose a stochastic message encoding/decoding procedure that makes a discrete communication channel mathematically equivalent to an analog channel with additive noise, through which gradients can be backpropagated. Additionally, we introduce an encryption step for use in noisy channels that forces channel noise to be message-independent, allowing us to compute unbiased derivative estimates even in the presence of unknown channel noise. To the best of our knowledge, this work presents the first differentiable communication learning approach that can compute unbiased derivatives through channels with unknown noise. We demonstrate the effectiveness of our approach in two example multi-robot tasks: a path finding and a collaborative search problem. There, we show that our approach achieves learning speed and performance similar to differentiable communication learning with real-valued messages (i. e. , unlimited communication bandwidth), while naturally handling more realistic real-world communication constraints. Content Areas: Multi-Agent Communication, Reinforcement Learning.

PDF Details

IROS Conference 2020 Conference Paper

Sparse Discrete Communication Learning for Multi-Agent Cooperation Through Backpropagation

  • Benjamin Freed
  • Rohan James
  • Guillaume Sartoretti
  • Howie Choset

Recent approaches to multi-agent reinforcement learning (MARL) with inter-agent communication have often overlooked important considerations of real-world communication networks, such as limits on bandwidth. In this paper, we propose an approach to learning sparse discrete communication through backpropagation in the context of MARL, in which agents are incentivized to communicate as little as possible while still achieving high reward. Building on top of our prior work on differentiable discrete communication learning, we develop a regularization-inspired message-length penalty term, that encourages agents to send shorter messages and avoid unnecessary communications. To this end, we introduce a variable-length message code that provides agents with a general means of modulating message length while keeping the overall learning objective differentiable. We present simulation results on a partially-observable robot navigation task, where we first show how our approach allows learning of sparse communication behavior while still solving the task. We finally demonstrate our approach can even learn an effective sparse communication behavior from demonstrations of an expert (potentially communication-free) policy.

Details

ICRA Conference 2019 Conference Paper

Workspace CPG with Body Pose Control for Stable, Directed Vision during Omnidirectional Locomotion

  • Samuel Shaw
  • Guillaume Sartoretti
  • Jake Olkin
  • William Paivine
  • Howie Choset

In this paper, we focus on the problem of directing the gaze of a vision system mounted to the body of a high-degree-of-freedom (DOF) legged robot for active perception deployments. In particular, we consider the case where the vision system is rigidly attached to the robot's body (i. e. , without any additional DOF between the vision system and robot body) and show how the supernumerary DOFs of the robot can be leveraged to allow independent locomotion and gaze control. Specifically, we augment a workspace central pattern generator (CPG) with omnidirectional capabilities by coupling it with a body pose control mechanism. We leverage the smoothing nature of the CPG framework to allow online adaptation of relevant locomotion parameters, and obtain a stable mid-level controller that translates desired gaze orientation and body velocity directly into joint angles. We validate our approach on an 18-DOF hexapod robot, in a series of indoor and outdoor trials, where the robot inspects an environmental feature or follows a pre-planned path relative to a visually-tracked landmark, demonstrating simultaneous locomotion and directed vision.

Details

ICRA Conference 2018 Conference Paper

Central Pattern Generator With Inertial Feedback for Stable Locomotion and Climbing in Unstructured Terrain

  • Guillaume Sartoretti
  • Samuel Shaw
  • Katie Lam
  • Naixin Fan
  • Matthew J. Travers
  • Howie Choset

Inspired by the locomotor nervous system of vertebrates, central pattern generator (CPG) models can be used to design gaits for articulated robots, such as crawling, swimming or legged robots. Incorporating sensory feedback for gait adaptation in these models can improve the locomotive performance of such robots in challenging terrain. However, many CPG models to date have been developed exclusively for open-loop gait generation for traversing level terrain. In this paper, we present a novel approach for incorporating inertial feedback into the CPG framework for the control of body posture during legged locomotion on steep, unstructured terrain. That is, we adapt the limit cycle of each leg of the robot with time to simultaneously produce locomotion and body posture control. We experimentally validate our approach on a hexapod robot, locomoting in a variety of steep, challenging terrains (grass, rocky slide, stairs). We show how our approach can be used to level the robot's body, allowing it to locomote at a relatively constant speed, even as terrain steepness and complexity prevents the use of an open-loop control strategy.

Details

ICRA Conference 2018 Conference Paper

Distributed Learning for the Decentralized Control of Articulated Mobile Robots

  • Guillaume Sartoretti
  • Yunfei Shi
  • William Paivine
  • Matthew J. Travers
  • Howie Choset

Decentralized control architectures, such as those conventionally defined by central pattern generators, independently coordinate spatially distributed portions of articulated bodies to achieve system-level objectives. State of the art distributed algorithms for reinforcement learning employ a different but conceptually related idea; independent agents simultaneously coordinating their own behaviors in parallel environments while asynchronously updating the policy of a system-or, rather, meta-level agent. This work, to the best of the authors' knowledge, is the first to explicitly explore the potential relationship between the underlying concepts in homogeneous decentralized control for articulated locomotion and distributed learning. We present an approach that leverages the structure of the asynchronous advantage actor-critic (A3C) algorithm to provide a natural framework for learning decentralized control policies on a single platform. Our primary contribution shows an individual agent in the A3C algorithm can be defined by an independently controlled portion of the robot's body, thus enabling distributed learning on a single platform for efficient hardware implementation. To this end, we show how the system is trained offline using hardware experiments implementing an autonomous decentralized compliant control framework. Our experimental results show that the trained agent outperforms the compliant control baseline by more than 40% in terms of steady progression through a series of randomized, highly cluttered evaluation environments.

Details

ICRA Conference 2018 Conference Paper

Proprioceptive-Inertial Autonomous Locomotion for Articulated Robots

  • Francesco Ruscelli
  • Guillaume Sartoretti
  • Junyu Nan
  • Zhixin Feng
  • Matthew J. Travers
  • Howie Choset

Inspired by the ability of animals to rely on proprioception and vestibular feedback to adapt their gait, we propose a modular framework for autonomous locomotion that relies on force sensing and inertial information. A first controller exploits anti-compliance, a new application of positive force feedback, to quickly react against obstacles upon impact. We hypothesize that, in situations where a robot experiences occasional impacts with the environment, anti-compliance can help negotiate unknown obstacles, similar to biological systems where positive feedback enables fast responses to external stimuli. A novel parallel controller, based on a bi-stable dynamical system, continuously adjusts the robot's direction of locomotion, and reverts it in reaction to major swerves. We present experimental results, demonstrating how our framework allows a snake robot to autonomously locomote through a row of unevenly-spaced obstacles. Finally, we extend our proprioceptive controller to legged locomotion, showing how a hexaprint robot can adapt its motion to climb over obstacles.

Details

ICRA Conference 2016 Conference Paper

Distributed planar manipulation in fluidic environments

  • Guillaume Sartoretti
  • Samuel Shaw
  • M. Ani Hsieh

We present a distributed control strategy that enables a swarm of autonomous surface vehicles (ASVs) to cooperatively grasp and manipulate a large floating object from an initial position and orientation to a desired final position and orientation. Given an initial set of random robot positions, our control strategy enables the team to synchronize their arrival times around the object. Analytical motion trajectories are then computed to enable the swarm to transport the object to the final desired position. We validate the proposed strategy in simulation and present experimental results to demonstrate the feasibility of the proposed strategy.

Details
v2026.09.13