Why Are Experts Correlated? Decomposing Correlations Between Judges

Abstract

We derive an analytic model of the inter-judge correlation as a function of five underlying parameters. Inter-cue correlation and the number of cues capture our assumptions about the environment, while differentiations between cues, the weights attached to the cues, and (un)reliability describe assumptions about the judges. We study the relative importance of, and interrelations between these five factors with respect to inter-judge correlation. Results highlight the centrality of the inter-cue correlation. We test the model's predictions with empirical data and illustrate its relevance. For example, we show that, typically, additional judges increase efficacy at a greater rate than additional cues.

Identifying Expertise to Extract the Wisdom of Crowds

“…Our results clearly suggest that it is primarily the model's ability to identify the experts to be positively weighted (or in other words, its ability to identify those members of the crowd who should be excluded) that is responsible for most of the model's improvement. This is not surprising, as the relative insensitivity of the model to departures from optimal weighting is well recognized in the literature (e.g., Broomell and Budescu 2009, Davis-Stober et al 2010, Dawes 1979. In fact, once the smallest subset of positive contributors is identified, there is a penalty associated with differential weighting (see Figure 6), and a simple unweighted mean of the carefully selected subset of judges provides the most accurate predictions.…”

“…The success of our approach is quite intuitive, once one realizes that judges are usually highly correlated (see Broomell and Budescu 2009) because they share many assumptions and/or have access to the same information. Consequently, crowds often behave like herds, as almost everyone expects certain events to happen (or not).…”

“…Lorenz et al (2011 demonstrated that even mild social influence can undermine the effect of WOC in simple estimation tasks. In fact, the higher the correlation between individual estimates, the more judges are necessary to achieve the same level of accuracy (e.g., Broomell and Budescu 2009, Clemen and Winkler 1986, Hogarth 1978.…”

“…We hypothesize that weights based on C j would outperform those based on the judge's absolute past performance (track record). The key intuition behind this prediction is that the forecasts of the various judges will typically be highly correlated (see Broomell and Budescu 2009). Thus, there will be many cases where almost everyone in the crowd will have very good scores and, conversely, cases where practically all the members of the crowd will perform poorly.…”

Optimal cue aggregation in the absence of criterion knowledge

“…Additionally, the learning of cue-criterion relationships (and subsequently accuracy in predicting the criterion) is sensitive to the structure of inter-cue correlation (Armelius & Armelius, 1976;Klayman, 1988; but also see Maines, 1996;Soll, 1999). Finally, in settings where cues themselves depend on a shared set of information sources, individuals are able to take into account the shared information structure and control for the resulting cue correlations, when predicting criterion values (Broomell & Budescu, 2009;Budescu & Rantilla, 2000;Budescu, Rantilla, Yu, & Karelitz, 2003).…”