Group Causal Policy Optimization for Post-Training Large Language Models

Recent advances in large language models (LLMs) have broadened their applicability across diverse tasks, yet specialized domains still require targeted post-training. Among existing methods, Group Relative Policy Optimization (GRPO) stands out for its efficiency, leveraging groupwise relative rewards while avoiding costly value function learning. However, GRPO treats candidate responses as independent, overlooking semantic interactions such as complementarity and contradiction. To address this challenge, we first introduce a Structural Causal Model (SCM) that reveals hidden dependencies among candidate responses induced by conditioning on a final integrated output, forming a collider structure. Then, our causal analysis leads to two insights: (1) projecting responses onto a causally-informed subspace improves prediction quality, and (2) this projection yields a better baseline than query-only conditioning. Building on these insights, we propose Group Causal Policy Optimization (GCPO), which integrates causal structure into optimization through two key components: a causally-informed reward adjustment and a novel KL-regularization term that aligns the policy with a causally-projected reference distribution. Comprehensive experimental evaluations on various benchmarks demonstrate that GCPO consistently surpasses existing methods.

Authors

• Ziyin Gu Institute of Software Chinese Academy of Sciences, University of the Chinese Academy of Sciences
• Jingyao Wang
• Ran Zuo Institute of Software Chinese Academy of Sciences, University of the Chinese Academy of Sciences, Communication University of China
• Chuxiong Sun Science and Technology on Integrated Information System Laboratory, Institute of Software Chinese Academy of Sciences State Key Laboratory of Intelligent Game
• Zeen Song
• Changwen Zheng Institute of Software, Chinese Academy of Sciences University of the Chinese Academy of Sciences
• Wenwen Qiang Science & Technology on Integrated Information System Laboratory, Institute of Software Chinese Academy of Sciences

Keywords

No keywords are indexed for this paper.