Probabilistic Coherence Weighting for Optimizing Expert Forecasts

Abstract

Methods for eliciting and aggregating expert judgment are necessary when decision-relevant data are scarce. Such methods have been used for aggregating the judgments of a large, heterogeneous group of forecasters, as well as the multiple judgments produced from an individual forecaster. This paper addresses how multiple related individual forecasts can be used to improve aggregation of probabilities for a binary event across a set of forecasters. We extend previous efforts that use probabilistic incoherence of an individual forecaster's subjective probability judgments to weight and aggregate the judgments of multiple forecasters for the goal of increasing the accuracy of forecasts. With data from two studies, we describe an approach for eliciting extra probability judgments to (i) adjust the judgments of each individual forecaster, and (ii) assign weights to the judgments to aggregate over the entire set of forecasters. We show improvement of up to 30% over the established benchmark of a simple equal-weighted averaging of forecasts. We also describe how this method can be used to remedy the “fifty–fifty blip” that occurs when forecasters use the probability value of 0.5 to represent epistemic uncertainty.

Use (and abuse) of expert elicitation in support of decision making for public policy

“…In laboratory studies, Karvetski et al (75) showed that by eliciting extra judgments to determine how coherent a judgment is, adjusting the resulting set of judgments to make them more coherent, and then weighting those adjusted judgment on the basis of original coherence, a significant improvement in performance could be achieved.…”

Crowdsourcing in business and management disciplines: an integrative literature review

“…Using machine learning techniques, Ghose et al (2012) proposed a random coefficient hybrid structural model for hotel ranking considering user behavior on social media and search engines. Based on two experiments over a crowdsourcing platform, a study by Karvetski et al (2013) argued that multiple related individual forecasts can be useful to improve aggregation of probabilities.…”

“…Karvetski et al (2013) Data from two events Establish a model that can be used to remedy that occurs when forecasters use the probability that represents epistemic uncertainty. Leimeister et al (2009) 32 samples from an online survey on a platform Find optimal incentive and motivation for an online idea competition.…”

Measuring inconsistency in probabilistic logic: rationality postulates and Dutch book interpretation

“…In this case, an inconsistency measure helps one to detect if a change approximates consistency or not. In other areas, inconsistency measures for probabilistic logic have found applications in merging conflicting opinions, leading to an increased predictive power [47,25], and in quantifying the incoherence of procedures from classical statistical hypothesis testing [41]. Example 1.1.…”