Jens Witkowski

AAAI Conference 2018 Conference Paper

Incentive-Compatible Forecasting Competitions

• Jens Witkowski
• Rupert Freeman
• Jennifer Vaughan
• David Pennock
• Andreas Krause

We consider the design of forecasting competitions in which multiple forecasters make predictions about one or more independent events and compete for a single prize. We have two objectives: (1) to award the prize to the most accurate forecaster, and (2) to incentivize forecasters to report truthfully, so that forecasts are informative and forecasters need not spend any cognitive effort strategizing about reports. Proper scoring rules incentivize truthful reporting if all forecasters are paid according to their scores. However, incentives become distorted if only the best-scoring forecaster wins a prize, since forecasters can often increase their probability of having the highest score by reporting extreme beliefs. Even if forecasters do report truthfully, awarding the prize to the forecaster with highest score does not guarantee that high-accuracy forecasters are likely to win; in extreme cases, it can result in a perfect forecaster having zero probability of winning. In this paper, we introduce a truthful forecaster selection mechanism. We lower-bound the probability that our mechanism selects the most accurate forecaster, and give rates for how quickly this bound approaches 1 as the number of events grows. Our techniques can be generalized to the related problems of outputting a ranking over forecasters and hiring a forecaster with high accuracy on future events.

AAAI Conference 2017 Conference Paper

Proper Proxy Scoring Rules

• Jens Witkowski
• Pavel Atanasov
• Lyle Ungar
• Andreas Krause

Proper scoring rules can be used to incentivize a forecaster to truthfully report her private beliefs about the probabilities of future events and to evaluate the relative accuracy of forecasters. While standard scoring rules can score forecasts only once the associated events have been resolved, many applications would benefit from instant access to proper scores. In forecast aggregation, for example, it is known that using weighted averages, where more weight is put on more accurate forecasters, outperforms simple averaging of forecasts. We introduce proxy scoring rules, which generalize proper scoring rules and, given access to an appropriate proxy, allow for immediate scoring of probabilistic forecasts. In particular, we suggest a proxy-scoring generalization of the popular quadratic scoring rule, and characterize its incentive and accuracy evaluation properties theoretically. Moreover, we thoroughly evaluate it experimentally using data from a large real world geopolitical forecasting tournament, and show that it is competitive with proper scoring rules when the number of questions is small.

AAAI Conference 2016 Conference Paper

A Geometric Method to Construct Minimal Peer Prediction Mechanisms

• Rafael Frongillo
• Jens Witkowski

Minimal peer prediction mechanisms truthfully elicit private information (e. g. , opinions or experiences) from rational agents without the requirement that ground truth is eventually revealed. In this paper, we use a geometric perspective to prove that minimal peer prediction mechanisms are equivalent to power diagrams, a type of weighted Voronoi diagram. Using this characterization and results from computational geometry, we show that many of the mechanisms in the literature are unique up to affine transformations, and introduce a general method to construct new truthful mechanisms.

AAAI Conference 2012 Conference Paper

A Robust Bayesian Truth Serum for Small Populations

• Jens Witkowski
• David Parkes

Peer prediction mechanisms allow the truthful elicitation of private signals (e. g. , experiences, or opinions) in regard to a true world state when this ground truth is unobservable. The original peer prediction method is incentive compatible for any number of agents n ≥ 2, but relies on a common prior, shared by all agents and the mechanism. The Bayesian Truth Serum (BTS) relaxes this assumption. While BTS still assumes that agents share a common prior, this prior need not be known to the mechanism. However, BTS is only incentive compatible for a large enough number of agents, and the particular number of agents required is uncertain because it depends on this private prior. In this paper, we present a robust BTS for the elicitation of binary information which is incentive compatible for every n ≥ 3, taking advantage of a particularity of the quadratic scoring rule. The robust BTS is the first peer prediction mechanism to provide strict incentive compatibility for every n ≥ 3 without relying on knowledge of the common prior. Moreover, and in contrast to the original BTS, our mechanism is numerically robust and ex post individually rational.

AAAI Conference 2011 Conference Paper

Incentive-Compatible Escrow Mechanisms

• Jens Witkowski
• Sven Seuken
• David Parkes

The most prominent way to establish trust between buyers and sellers on online auction sites are reputation mechanisms. Two drawbacks of this approach are the reliance on the seller being long-lived and the susceptibility to whitewashing. In this paper, we introduce so-called escrow mechanisms that avoid these problems by installing a trusted intermediary which forwards the payment to the seller only if the buyer acknowledges that the good arrived in the promised condition. We address the incentive issues that arise and design an escrow mechanism that is incentive compatible, efficient, interim individually rational and ex ante budget-balanced. In contrast to previous work on trust and reputation, our approach does not rely on knowing the sellers’ cost functions or the distribution of buyer valuations.

IJCAI Conference 2011 Conference Paper

Trust Mechanisms for Online Systems (Extended Abstract)

• Jens Witkowski

The most prominent way to establish trust in online markets such as eBay are reputation systems that publish buyer feedback about a seller's past behavior. These systems, however, critically rely on assumptions that are rarely met in real-world marketplaces: first, it is assumed that there are no reporting costs and no benefits from lying so that buyers honestly report their private experiences. Second, it is assumed that every seller is long-lived, i. e. will continue to trade on the marketplace indefinitely and, third, it is assumed that sellers cannot whitewash, i. e. create new accounts once an old one is ran down. In my thesis, I address all of these assumptions and design incentive-compatible trust mechanisms that do not rely on any of the aforementioned assumptions. Moreover, I focus on designs that minimize common knowledge assumptions with respect to the players' valuations, costs and beliefs.

UAI Conference 2010 Conference Paper

Truthful Feedback for Sanctioning Reputation Mechanisms

• Jens Witkowski

For product rating environments, similar to that of Amazon Reviews, it has been shown that the truthful elicitation of feedback is possible through mechanisms which pay buyer reports contingent on the reports of other buyers. We study whether similar mechanisms can be designed for reputation mechanisms at online auction sites where the buyers’ experiences are partially determined by a strategic seller. We show that this is impossible for the basic setting. However, introducing a small prior belief that the seller is a cooperative commitment player leads to a payment scheme with a truthful perfect Bayesian equilibrium.

IJCAI Conference 2009 Conference Paper

• Jens Witkowski

Recently, online reputation mechanisms have been proposed that reward agents for honest feedback about products and services with fixed quality. Many real-world settings, however, are inherently dynamic. As an example, consider a web service that wishes to publish the expected download speed of a file mirrored on different server sites. In contrast to the models of Miller, Resnick and Zeckhauser and of Jurca and Faltings, the quality of the service (e. g. , a server’s available bandwidth) changes over time and future agents are solely interested in the present quality levels. We show that hidden Markov models (HMM) provide natural generalizations of these static models and design a payment scheme that elicits honest reports from the agents after they have experienced the quality of the service.