GeWu: A Culturally-Grounded Chinese Benchmark for Multi-Stage Social Bias Evaluation in Large Language Models

With the rapid deployment of Chinese large language models (LLMs), culturally-grounded bias evaluation remains understudied due to the dominance of English benchmarks and simplistic Chinese scenarios. To address this, we propose GeWu, a comprehensive benchmark featuring a culturally-aware dataset of 60,192 questions spanning 14 social groups with fine-grained Chinese contexts, significantly exceeding existing resources in breadth and depth. Our two-stage evaluation first quantifies bias via multiple-choice questions using a novel probability-based scoring mechanism to sensitively capture bias tendencies, distilling high-bias scenarios into GeWu-1K. This refined subset then enables multi-turn dialogue evaluations for in-depth analysis under realistic conditions. Experiments reveal that GeWu effectively exposes social biases in state-of-the-art Chinese LLMs, with 13.93% of scenarios eliciting universal bias across all models. This highlights persistent challenges and provides actionable insights for bias mitigation in Chinese contexts.

Authors

• Yi Lin College of Computer Science, Sichuan University, Chengdu 610000, China
• Ziyi Zhou Beihang University
• Jiashi Gao Southern University of Science and Technology
• Xinwei Guo Southern University of Science and Technology
• Jiaxin Zhang South China University of Technology
• Haiyan Wu University of Macau
• Xin Yao Lingnan University
• Xuetao Wei Southern University of Science and Technology

Keywords

No keywords are indexed for this paper.