ICML Conference 2025 Conference Paper
Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMs
- Yinong Oliver Wang
- Nivedha Sivakumar
- Falaah Arif Khan
- Katherine Metcalf
- Adam Golinski
- Natalie Mackraz
- Barry-John Theobald
- Luca Zappella
The recent rapid adoption of large language models (LLMs) highlights the critical need for benchmarking their fairness. Conventional fairness metrics, which focus on discrete accuracy-based evaluations (i. e. , prediction correctness), fail to capture the implicit impact of model uncertainty (e. g. , higher model confidence about one group over another despite similar accuracy). To address this limitation, we propose an uncertainty-aware fairness metric, UCerf, to enable a fine-grained evaluation of model fairness that is more reflective of the internal bias in model decisions. Furthermore, observing data size, diversity, and clarity issues in current datasets, we introduce a new gender-occupation fairness evaluation dataset with 31, 756 samples for co-reference resolution, offering a more diverse and suitable benchmark for modern LLMs. Combining our metric and dataset, we provide insightful comparisons of eight open-source LLMs. For example, Mistral-8B exhibits suboptimal fairness due to high confidence in incorrect predictions, a detail overlooked by Equalized Odds but captured by UCerF. Overall, this work provides a holistic framework for LLM evaluation by jointly assessing fairness and uncertainty, enabling the development of more transparent and accountable AI systems.