Deep (Predictive) Discounted Counterfactual Regret Minimization

Counterfactual regret minimization (CFR) is a family of algorithms for effectively solving imperfect-information games. To enhance CFR's applicability in large games, researchers use neural networks to approximate its behavior. However, existing methods are mainly based on vanilla CFR and struggle to effectively integrate more advanced CFR variants. In this work, we propose an efficient model-free neural CFR algorithm, overcoming the limitations of existing methods in approximating advanced CFR variants. At each iteration, it collects variance-reduced sampled advantages based on a value network, fits cumulative advantages by bootstrapping, and applies discounting and clipping operations to simulate the update mechanisms of advanced CFR variants. Experimental results show that, compared with model-free neural algorithms, it exhibits faster convergence in typical imperfect-information games and demonstrates stronger adversarial performance in a large poker game.

Authors

• Hang Xu Huawei Noah's Ark Lab
• Kai Li
• Haobo Fu Tencent AI Lab
• Qiang Fu Tencent AI Lab
• Junliang Xing Tsinghua University
• Jian Cheng Institute of Automation, Chinese Academy of Sciences School of Future Technology, University of Chinese Academy of Sciences AiRiA

Keywords

No keywords are indexed for this paper.