Structured Agent Distillation for Large Language Model Agents

Largelanguagemodels(LLMs)exhibitstrongcapabilitiesasdecisionmaking agents by interleaving reasoning and actions, as seen in ReAct-style frameworks. Yet, their practical deployment is constrained by high inference costs and large model sizes. We propose Structured Agent Distillation, the first framework to distill a ReAct-based LLM agent into a smaller model while preserving both reasoning fidelity and action consistency. Our method introduces a structured, span-level distillation strategy that explicitly segments trajectories into reasoning and action spans, enabling fine-grained alignment beyond standard token-level imitation. Unlike other advanced distillation methods, Our method segments trajectories into [REASON] and [ACT] spans, applying segmentspecific losses to align each component with the teacher’s behavior. This structure-aware supervision enables compact agents to better Corresponding Authors: Pu Zhao, Hao Tang. This work is licensed under a Creative Commons Attribution International 4. 0 License. Proc. of the 25th International Conference on Autonomous Agents and Multiagent Systems (AAMAS 2026), C. Amato, L. Dennis, V. Mascardi, J. Thangarajah (eds.), May 25 – 29, 2026, Paphos, Cyprus. © 2026 International Foundation for Autonomous Agents and Multiagent Systems (www. ifaamas. org). https: //doi. org/10. 65109/OLHJ8062 replicate the teacher’s decision process. Experiments on ALFWorld, HotPotQA-ReAct, and WebShop show that our approach consistently outperforms token-level and imitation learning baselines, achieving significant compression with minimal performance drop. Scaling and ablation results further highlight the importance of span-level alignment for efficient and deployable agents. We will release code upon acceptance.

Authors

• Jun Liu Carnegie Mellon University
• Zhenglun Kong Harvard University
• Peiyan Dong MIT
• Changdi Yang Northeastern University
• Tianqin Li Carnegie Mellon University
• Yanyue Xie Northeastern University
• Yifan Gong Adobe Research
• Xuan Shen Northeastern University
• Pu Zhao Northeastern University
• Hao Tang Carnegie Mellon University
• Geng Yuan University of Georgia
• Wei Niu University of Georgia
• Wenbin Zhang Florida International University
• Xue Lin Northeastern University
• Dong Huang Carnegie Mellon University
• Yanzhi Wang Northeastern University

Keywords

• Agent Distillation
• Span-Level Alignment
• Reasoning and Action Segmentation
• Large Language Models (LLMs)