Validating the Contribution-Weighted Model: Robustness and Cost-Benefit Analyses

Abstract

We use results from a multiyear, geopolitical forecasting tournament to highlight the ability of the contribution weighted model [Budescu DV, Chen E (2015) Identifying expertise to extract the wisdom of crowds. Management Sci. 61(2):267–280] to capture and exploit expertise. We show that the model performs better when judges gain expertise from manipulations such as training in probabilistic reasoning and collaborative interaction from serving on teams. We document the model’s robustness using probability judgments from early, middle, and late phases of the forecasting period and by showing its strong performance in the presence of hypothetical malevolent forecasters. The model is highly cost-effective: it operates well, even with random attrition, as the number of judges shrinks and information on their past performance is reduced.

Using meta-predictions to identify experts in the crowd when past performance is unknown

“…In line with Budescu and Chen [6] and Chen et al [17], we compare the performance of the MPW algorithm and other probabilistic aggregation approaches using a transformed Brier score:…”

A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment

“…On the one hand, there is evidence that the performance of the UWM is often relatively close to that of a comparable benchmark using a non-equal weighting (e.g., Clemen and Winkler 1986;Einhorn et al 1977;Flandoli et al 2011). On the other hand, studies also support the superior performance of weighting-based algorithms (Cooke and Goossens 2008;Hammitt and Zhang 2013;Budescu and Chen 2015;Chen et al 2016). Consequently, aggregation algorithms leave room for exploration.…”

“…Thinking of an unbiased expert judgment as the true value plus a random error (as done by Hammitt and Zhang, 2013), according to the law of large numbers, an increasing number of experts will stabilize the aggregated judgment around the true value and decrease variance (Einhorn et al 1977). The effect of increasing aggregation performance with increasing crowd size has been shown analytically (Hogarth, 1978), empirically (Chen et al 2016), and via simulation (Wagner and Vinaimont 2010). However, it is important to assume that the increase in crowd size originates from randomly selected experts and not from specifically characterized experts (e.g., unqualified ones).…”

“…Based on the UWM, other algorithms have been developed that weight experts by including a measure of the experts' past performance. The Brier Weighted Model (Budescu and Chen 2015;Chen et al 2016), e.g., computes the Brier Score (Brier 1950) for every expert, averaging it over all historical events. Based on this, weights (w n ) are allocated to the experts.…”

The Wisdom of a Kalman Crowd

“…Other rules grant access to information about prior performances (King et al, 2012), while some combine estimates by subsets of people and ignore those made by others (Budescu and Chen, 2015;Mannes et al, 2014). The latest rules in that last category have done particularly well (Budescu and Chen, 2015;Chen et al, 2016). More specifically, the so-called Contribution Weighted Model (CWM) has beaten many other rules not only because it has an algorithm for finding subsets of strong forecasters, but also because it puts different weights on their estimates using relative measures (Budescu and Chen, 2015;Chen et al, 2016) of how they performed in the past.…”