ResMAS: Resilience Optimization in LLM-based Multi-agent Systems

Large Language Model-based Multi-Agent Systems (LLM-based MAS), where multiple LLM agents collaborate to solve complex tasks, have shown impressive performance in many areas. However, MAS are typically distributed across different devices or environments, making them vulnerable to perturbations such as agent failures. While existing works have studied the adversarial attacks and corresponding defense strategies, they mainly focus on reactively detecting and mitigating attacks after they occur rather than proactively designing inherently resilient systems. In this work, we study the resilience of LLM-based MAS under perturbations and find that both the communication topology and prompt design significantly influence system resilience. Motivated by these findings, we propose ResMAS: a two-stage framework for enhancing MAS resilience. First, we train a reward model to predict the MAS’s resilience, based on which we train a topology generator to automatically design resilient topology for specific tasks through reinforcement learning. Second, we introduce a topology-aware prompt optimization method that refines each agent’s prompt based on its connections and interactions with other agents. Extensive experiments across a range of tasks show that our approach substantially improves MAS resilience under various constraints. Moreover, our framework demonstrates strong generalization ability to new tasks and models, highlighting its potential for building resilient MASs.

Authors

• Zhilun Zhou Tsinghua University
• Zihan Liu Zhejiang University, Hangzhou AI Lab, Research Center for Industries of the Future, Westlake University
• Jiahe Liu Tsinghua University
• Qingyu Shao Tsinghua University
• Yihan Wang Academy of Mathematics and Systems Science, Chinese Academy of Sciences University of Chinese Academy of Sciences
• Kun Shao Huawei
• Depeng Jin BNRist, Tsinghua University
• Fengli Xu

Keywords

No keywords are indexed for this paper.