Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment

We consider the issue of strategic behaviour in various peerassessment tasks, including peer grading of exams or homeworks and peer review in hiring or promotions. When a peerassessment task is competitive (e. g. , when students are graded on a curve), agents may be incentivized to misreport evaluations in order to improve their own final standing. Our focus is on designing methods for detection of such manipulations. Specifically, we consider a setting in which agents evaluate a subset of their peers and output rankings that are later aggregated to form a final ordering. In this paper, we investigate a statistical framework for this problem and design a principled test for detecting strategic behaviour. We prove that our test has strong false alarm guarantees and evaluate its detection ability in practical settings. For this, we design and conduct an experiment that elicits strategic behaviour from subjects and release a dataset of patterns of strategic behaviour that may be of independent interest. We use this data to run a series of real and semi-synthetic evaluations that reveal a strong detection power of our test.

Authors

• Ivan Stelmakh
• Nihar B. Shah
• Aarti Singh

Keywords

No keywords are indexed for this paper.