DEPO: Dual-Efficiency Preference Optimization for LLM Agents

Recent advances in large language models (LLMs) have greatly improved their reasoning and decision-making abilities when deployed as agents. Richer reasoning, however, often comes at the cost of longer chain of thought (CoT), hampering interaction efficiency in real-world scenarios. Nevertheless, there still lacks systematic definition of LLM‑Agent efficiency, hindering targeted improvements. To this end, we introduce dual‑efficiency, comprising (i) step-level efficiency, which minimizes tokens per step, and (ii) trajectory-level efficiency, which minimizes the number of steps to complete a task. Building on this definition, we propose DEPO, a dual-efficiency preference‑based optimization method that jointly rewards succinct responses and fewer action steps. Experiments on WebShop and BabyAI show that DEPO cuts token usage by up to 60.9% and steps by up to 26.9%, while achieving up to a 29.3% improvement in task performance. DEPO also generalizes to three out-of-domain math benchmarks and retains its efficiency gains when trained on only 25% of the data.

Authors

• Sirui Chen Renmin University of China
• Mengshi Zhao The University of Hong Kong
• Lei Xu Institute of Cyber Science and Technology, Shanghai Jiao Tong University, Shanghai, China Shanghai Key Laboratory of Integrated Administration Technologies for Information Security, Shanghai, China
• Yuying Zhao
• Beier Zhu Nanyang Technological University
• Hanwang Zhang Nanyang Technological University
• Shengjie Zhao
• Chaochao Lu Shanghai AI Laboratory

Keywords

No keywords are indexed for this paper.