A Robust Bayesian Truth Serum for Non-Binary Signals

Radanovic, Goran; Faltings, Boi

doi:10.1609/aaai.v27i1.8677

Cited by 51 publications

(36 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One difference to peer prediction is that, at the heart of proper and proxy scoring is the elicitation and scoring of probabilistic forecasts, whereas peer prediction is primarily concerned with the elicitation of informative signals. While Bayesian Truth Serum mechanisms (Prelec 2004;Witkowski and Parkes 2012;Radanovic and Faltings 2013) also elicit a probabilistic report, the probabilistic report is only a means towards eliciting the signal. The proxy scoring framework can be viewed as providing a link between the literature on proper scoring rules and peer prediction.…”

Section: Introductionmentioning

confidence: 99%

Proper Proxy Scoring Rules

Witkowski

Atanasov²,

Ungar

et al. 2017

AAAI

View full text Add to dashboard Cite

Proper scoring rules can be used to incentivize a forecaster to truthfully report her private beliefs about the probabilities of future events and to evaluate the relative accuracy of forecasters. While standard scoring rules can score forecasts only once the associated events have been resolved, many applications would benefit from instant access to proper scores. In forecast aggregation, for example, it is known that using weighted averages, where more weight is put on more accurate forecasters, outperforms simple averaging of forecasts. We introduce proxy scoring rules, which generalize proper scoring rules and, given access to an appropriate proxy, allow for immediate scoring of probabilistic forecasts. In particular, we suggest a proxy-scoring generalization of the popular quadratic scoring rule, and characterize its incentive and accuracy evaluation properties theoretically. Moreover, we thoroughly evaluate it experimentally using data from a large real world geopolitical forecasting tournament, and show that it is competitive with proper scoring rules when the number of questions is small.

show abstract

Section: Introductionmentioning

confidence: 99%

Proper Proxy Scoring Rules

Witkowski

Atanasov²,

Ungar

et al. 2017

AAAI

View full text Add to dashboard Cite

show abstract

“…This problem is called information elicitation without verification (IEWV) (Waggoner and Chen 2014). A class of mechanisms, collectively called peer prediction, has been developed for the IEWV problem (Prelec 2004;Miller, Resnick, and Zeckhauser 2005;Jurca and Faltings 2007;2009;Witkowski and Parkes 2012a;2012b;Radanovic and Faltings 2013). In peer prediction, an agent is rewarded according to how his answer compares with those of his peers and the reward rules are designed so that everyone truthfully reporting their information is a game-theoretic equilibrium.…”

Section: Introductionmentioning

confidence: 99%

Sequential Peer Prediction: Learning to Elicit Effort using Posted Prices

Liu

2017

AAAI

View full text Add to dashboard Cite

Peer prediction mechanisms are often adopted to elicit truthful contributions from crowd workers when no ground-truth verification is available. Recently, mechanisms of this type have been developed to incentivize effort exertion, in addition to truthful elicitation. In this paper, we study a sequential peer prediction problem where a data requester wants to dynamically determine the reward level to optimize the trade-off between the quality of information elicited from workers and the total expected payment. In this problem, workers have homogeneous expertise and heterogeneous cost for exerting effort, both unknown to the requester. We propose a sequential posted-price mechanism to dynamically learn the optimal reward level from workers' contributions and to incentivize effort exertion and truthful reporting. We show that (1) in our mechanism, workers exerting effort according to a non-degenerate threshold policy and then reporting truthfully is an equilibrium that returns highest utility for every worker, and (2) The regret of our learning mechanism w.r.t. offering the optimal reward (price) is upper bounded by Õ(T{3/4) where T is the learning horizon. We further show the power of our learning approach when the reports of workers do not necessarily follow the game-theoretic equilibrium.

show abstract

“…The major shortcoming of the classical peer prediction method with regard to practical applications is that it requires too much common knowledge. Bayesian Truth Serum mechanisms (Prelec 2004;Witkowski and Parkes 2012a;Radanovic and Faltings 2013) relax these common knowledge assumptions but require participants to report a probability distribution in addition to the actual information that is to be elicited. That is, they are not minimal.…”

Section: Introductionmentioning

confidence: 99%

A Geometric Method to Construct Minimal Peer Prediction Mechanisms

Frongillo

Witkowski

2016

AAAI

View full text Add to dashboard Cite

Minimal peer prediction mechanisms truthfully elicit private information (e.g., opinions or experiences) from rational agents without the requirement that ground truth is eventually revealed. In this paper, we use a geometric perspective to prove that minimal peer prediction mechanisms are equivalent to power diagrams, a type of weighted Voronoi diagram. Using this characterization and results from computational geometry, we show that many of the mechanisms in the literature are unique up to affine transformations, and introduce a general method to construct new truthful mechanisms.

show abstract

A Robust Bayesian Truth Serum for Non-Binary Signals

Cited by 51 publications

References 13 publications

Proper Proxy Scoring Rules

Proper Proxy Scoring Rules

Sequential Peer Prediction: Learning to Elicit Effort using Posted Prices

A Geometric Method to Construct Minimal Peer Prediction Mechanisms

Contact Info

Product

Resources

About