Token-level Advantage Policy Optimization from Negative Feedback in Multi-Turn Agents

Trainingmulti-turnagentsforcomplextasksischallengedbysparse rewards. Existing methods are inefficient: they either learn exclusively from successes, discarding valuable failure data, or require rigid win-loss pairs, limiting data utilization. We propose Tokenlevel Advantage Policy Optimization (TAPO), a flexible, pair-free methodthatleveragesalltrajectories. TAPOtranslatesatrajectory’s terminal reward into token-level advantages, effectively reinforcing the entire sequence of actions in successful trajectories while penalizing those in failed ones. Furthermore, TAPO concentrates updates on high-entropy tokens, which represent pivotal moments of model uncertainty and are thus crucial for efficient exploration and policy improvement. As a post-training optimization, TAPO substantially boostsabaselineSFTagent’saveragescorefrom74. 2to89. 4(+20. 5% relative) on three challenging multi-turn benchmarks, outperformingRFTandnegative-awarebaselinesanddemonstratingconsistent gains in both seen and unseen settings. The code is available at https: //github. com/Sunrepe/TAPO.

Authors

• Xufeng Zhou Institute of Automation, Chinese Academy of Sciences & School of Artificial Intelligence, University of Chinese Academy of Sciences
• Linjing Li Institute of Automation, Chinese Academy of Sciences & Wenge Technology Co. , Ltd.
• Daniel Dajun Zeng Institute of Automation, Chinese Academy of Sciences & School of Artificial Intelligence, University of Chinese Academy of Sciences

Keywords

• Multi-turn agents
• Token-level Policy optimization
• Entropy-guided Learning
• GAAI