Does Calibration Affect Human Actions?

Calibration has been proposed as a way to enhance the reliability and adoption of machine learning classifiers. We study a particular aspect of this proposal: how does calibrating a classification model affect the decisions made by non-expert humans consuming the model’s predictions? We performed a Human–Computer Interaction (HCI) experiment to assess the effect of calibration on (i) trust in the model and (ii) the correlation between decisions and predictions. Wealsoproposefurthercorrectionstothereportedcalibrated scores based on Kahneman and Tversky’s prospect theory from behavioral economics, and study the effect of these corrections on trust and decision-making. We found that calibration alone was not sufficient: applying the prospect-theory–based correction is crucial for increasing the correlation between human decisions and the model predictions. While this increased correlation suggests higher trust in the model, responses to “Do you trust the model more? ” are unaffected by the method used. For the full paper, see: https: //arxiv. org/abs/2508. 18317

Authors

• Meir Nizri Department of Data Science and AI, Ariel University
• Amos Azaria Department of Data Science and AI, Ariel University
• Chirag Gupta Carnegie Mellon University
• Noam Hazon Department of Data Science and AI, Ariel University

Keywords

• Model Calibration
• Prospect Theory
• Human-Agent Interaction