The Wisdom of Multiple Guesses

Abstract

The "wisdom of crowds" dictates that aggregate predictions from a large crowd can be surprisingly accurate, rivaling predictions by experts. Crowds, meanwhile, are highly heterogeneous in their expertise. In this work, we study how the heterogeneous uncertainty of a crowd can be directly elicited and harnessed to produce more efficient aggregations from a crowd, or provide the same efficiency from smaller crowds. We present and evaluate a novel strategy for eliciting sufficient information about an individual's uncertainty: allow individuals to make multiple simultaneous guesses, and reward them based on the accuracy of their closest guess. We show that our multiple guesses scoring rule is an incentive-compatible elicitation strategy for aggregations across populations under the reasonable technical assumption that the individuals all hold symmetric log-concave belief distributions that come from the same location-scale family. We first show that our multiple guesses scoring rule is strictly proper for a fixed set of quantiles of any log-concave belief distribution. With properly elicited quantiles in hand, we show that when the belief distributions are also symmetric and all belong to a single location-scale family, we can use interquantile ranges to furnish weights for certainty-weighted crowd aggregation. We evaluate our multiple guesses framework empirically through a series of incentivized guessing experiments on Amazon Mechanical Turk, and find that certainty-weighted crowd aggregations using multiple guesses outperform aggregations using single guesses without certainty weights.

What interventions can decrease or increase belief polarisation in a population of rational agents?

“…Instead, in our simulations, whenever an information source needed to know the ground truth in order to determine what signal to broadcast, it had to estimate the ground truth by taking the median value of the initial beliefs of the agents. We used the median, as opposed to the mean, as the median is more robust to bias (Ugander & Guestrin, 2015). Even so, because there were relatively few agents and because the initial beliefs of the agents were biased, this meant that the estimate of the ground truth did not always accurately reflect the actual ground truth.…”

“…The assumption that the ground truth is not known, while restrictive, is a necessary reflection of real life. Despite this assumption, one can still estimate the ground truth by taking the median of the initial beliefs of all the agents (Ugander & Guestrin, 2015). While this estimate may not perfectly correspond to the ground truth, it will typically be closer to the ground truth than the estimates of most of the agents (Surowiecki, 2004).…”

Ballpark Crowdsourcing

“…Importantly, virtually all of these methods require aggregation of crowd guesses. A large body of work in crowdsourcing for machine learning and "wisdom of the crowds" (WoC) deals with aggregating human guesses [7,13,17,24,25,28]. The objective of these methods is typically to obtain accurate point estimates of some quantity.…”

An Overview of Applications of Proper Scoring Rules

“…Crowdsourcing Jurca and Faltings (2009), Faltings et al (2012), Kamar and Horvitz (2012), Zhang et al (2012), Ray et al (2013), Sakurai et al (2013), Cao et al (2014), Faltings et al (2014), Oka et al (2014), Sakurai et al (2015), Ugander et al (2015) Economics von Holstein (1972), O'Carroll (1977, Yates et al (1991), Muradoglu and Onkal (1994), Hirtle and Lopez (1999), Lopez (2001), Casillas-Olvera and Bessler (2006), Gschlößl and Czado (2007), Carvalho and Larson (2010, Diebold and Mariano (2012), Lad et al (2012), Johnstone et al 2013Education Bickel (2007Bickel ( , 2010 Electronic commerce Miller et al (2005), Gerding et al (2010), Cai et al (2013), Radanovic and Faltings (2015) Energy Woo et al (1998), , Pinson et al (2007) 2015Health/Medicine Dolan et al (1986), Spiegelhalter (1986), Linnet (1988Linnet ( , 1989, Spiegelhalter et al (1990), Bernardo and Muñoz (1993), Winkler and Poses (1993), Madigan et al (1995),…”