Proper Proxy Scoring Rules

Abstract

Proper scoring rules can be used to incentivize a forecaster to truthfully report her private beliefs about the probabilities of future events and to evaluate the relative accuracy of forecasters. While standard scoring rules can score forecasts only once the associated events have been resolved, many applications would benefit from instant access to proper scores. In forecast aggregation, for example, it is known that using weighted averages, where more weight is put on more accurate forecasters, outperforms simple averaging of forecasts. We introduce proxy scoring rules, which generalize proper scoring rules and, given access to an appropriate proxy, allow for immediate scoring of probabilistic forecasts. In particular, we suggest a proxy-scoring generalization of the popular quadratic scoring rule, and characterize its incentive and accuracy evaluation properties theoretically. Moreover, we thoroughly evaluate it experimentally using data from a large real world geopolitical forecasting tournament, and show that it is competitive with proper scoring rules when the number of questions is small.

Forecast Aggregation via Peer Prediction

“…Our construction of PAS is derived from a family of mechanisms collectively called peer prediction. The term peer prediction was coined up by (Miller, Resnick, and Zeckhauser 2005) and the literature has been further developed by a series of studies (e.g., Prelec 2004;Miller, Resnick, and Zeckhauser 2005;Shnayder et al 2016;Radanovic, Faltings, and Jurca 2016;Witkowski et al 2017;Kong 2020).…”

“…But they require to know the ground truth of predicted events. Surrogate scoring rules (SSR) and proxy scoring rules (PSR) (Witkowski et al 2017) are two peer prediction mechanisms that try to recover the SPSR from participants' reports, thus providing two methods to estimate the prediction accuracy of agents in the minimal information setting. Both mechanisms estimate a proxy of ground truth from participants' forecasts and assess their forecasts against this proxy.…”

“…In constrast to SSR, PSR directly apply SPSR S(•) to an agent's forecast against a proxy Ŷi of the ground truth to obtain the reward score, i.e., R PSR i,j (p i,j , Ŷi ) = S(p i,j , Ŷi ). Witkowski et al (2017) showed that as long as the proxy Ŷi is unbiased to the ground truth, the proxy scoring rule gives an positive affine transformation of S(•), maintaining the incentive property. In practice, Witkowski et al (2017) recommended using an extremized mean prediction as the proxy when there is no explicit unbiased proxy of ground truth available.…”

“…In this paper, we explore the use of five recently proposed peer prediction mechanisms (Radanovic, Faltings, and Jurca 2016;Shnayder et al 2016;Witkowski et al 2017; Chen 2020; Kong 2020) as PAS. After showing their theoretical properties in recovering the forecasters' true performance, we thoroughly examine the empirical performance of PAS-aided aggregators built upon them.…”

Information Gathering With Peers: Submodular Optimization With Peer-Prediction Constraints

“…Instead of evaluating data providers against trusted reports, Dasgupta and Ghosh (2013), Jurca and Faltings (2011), Radanovic, Faltings, and Jurca (2016), Shnayder et al (2016), Witkowski et al (2017), and Baillon (2017), propose peerprediction mechanisms for incentivizing distributed information sources. Peer-prediction mechanisms reward data providers by measuring consistency among their reportsthus, if a data provider believes that others are honest, she is also incentivized to report truthfully.…”