Averaging probability judgments: Monte Carlo analyses of asymptotic diagnostic value

Abstract

Wallsten et al. (1997) developed a general framework for assessing the quality of aggregated probability judgments. Within this framework they presented a theorem regarding the effects of pooling multiple probability judgments regarding unique binary events. The theorem states that under reasonable conditions, and assuming conditional pairwise independence of the judgments, the average probability estimate is asymptotically perfectly diagnostic of the true event state as the number of estimates pooled goes to in®nity. The purpose of the present study was to examine by simulation (1) the rate of convergence of averaged judgments to perfect diagnostic value under various conditions and (2) the robustness of the theorem to violations of its assumption that the covert probability judgments are conditionally pairwise independent. The results suggest that while the rate of convergence is sensitive to violations of the conditional pairwise independence, the asymptotic properties remain relatively robust under a large variety of conditions. The practical implications of these results are discussed.

Decisions from experience: Why small samples?

“…Moreover, although a person's ability to choose the higher expected value gamble was diminished with small samples, our analysis showed that small samples are not ruinous to choosing the higher expected value gamble. Johnson, Budescu, and Wallsten (2001) reported an analogous result when aggregating probability judgments to form a new probability judgment: judges can be quite accurate when averaging very few judgments. Before we describe testable implications of the amplification effect, let us briefly turn to how the difference between options relates to difficulty of choice and to choice strategies.…”

Bias Blind Spot: Structure, Measurement, and Consequences

“…As we have demonstrated, awareness of one's vulnerability to bias is an important antecedent of openness to advice that in turn affects decision quality. Research maintains that integrating advice from external sources improves decision making (Larrick and Soll 2006, Johnson et al 2001, Budescu and Rantilla 2000, Yaniv 2004; see Bonaccio and Dalal 2006 for a review). People high in bias blind spot, for example, may be particularly likely to ignore the advice of peers or experts when engaging in financial or medical decision making and require alternative forms of guidance to improve the quality of their decisions.…”

Aggregation of opinions based on correlated cues and advisors

“…A general conclusion from this work is that some form of averaging describes very well the observed behavior in most cases (e.g., Clemen, 1989;Fischer & Harvey, 1999;Wallsten, Budescu, & Tsao, 1997). The accuracy of the average opinion increases monotonically as a function of the number of advisors, but at a diminishing rate that depends on the inter-judge correlation (e.g., Ariely et al, 2000;Clemen & Winkler, 1985;Hogarth, 1978;Johnson, Budescu, & Wallsten, 2001;Wallsten & Diederich, 2001). Although under certain circumstances, averaging is also normatively optimal, people fail to appreciate the benefits of this simple aggregation rule (Larrick & Soll, 2006).…”