Identifying and Cultivating Superforecasters as a Method of Improving Probabilistic Predictions

Abstract

Across a wide range of tasks, research has shown that people make poor probabilistic predictions of future events. Recently, the U.S. Intelligence Community sponsored a series of forecasting tournaments designed to explore the best strategies for generating accurate subjective probability estimates of geopolitical events. In this article, we describe the winning strategy: culling off top performers each year and assigning them into elite teams of superforecasters. Defying expectations of regression toward the mean 2 years in a row, superforecasters maintained high accuracy across hundreds of questions and a wide array of topics. We find support for four mutually reinforcing explanations of superforecaster performance: (a) cognitive abilities and styles, (b) task-specific skills, (c) motivation and commitment, and (d) enriched environments. These findings suggest that superforecasters are partly discovered and partly created-and that the high-performance incentives of tournaments highlight aspects of human judgment that would not come to light in laboratory paradigms focused on typical performance.

Forecasts or fortune-telling: When are expert judgements of safety risk valid?

“…Mellers et al report on a large scale experiment in the identification and development of superforecasters (Mellers et al, 2015). The experiment made use of a forecasting competition to identify individuals who were skilled at real-world predictions.…”

“…Research on expert elicitation that suggests that particular modes of thinking result in better estimates (Mellers et al, 2015). However, elicitation research is itself mostly only validated for non-forecast probabilistic judgements.…”

“…However, there is a possibility that discussion between experts will identify errors in reasoning, and will enhance the information available to each expert (Mellers et al, 2015;Wright & Rowe, 2011) The counterargument is that groups are predisposed to work towards consensus, rather than making small adjustments to the individual positions (Solomon, 2006). This phenomenon groupthink can result in the group selecting the most confident or authoritative opinion.…”

Confidence Calibration in a Multi-year Geopolitical Forecasting Competition

“…Mellers et al (2015a) explore the profiles of individual forecasters using dispositional, situational, and behavioral variables. In another paper, Mellers et al (2015b) document the performance of the most accurate performers, known as superforecasters, and identify reasons for their success.…”

Expertise, credibility of system forecasts and integration methods in judgmental demand forecasting

“…The scale is based on peer ratings, and therefore is subject to power biases in organizations; however, it is clearly an improvement on the widely used, largely unstructured methods of selecting experts based on public recognition or co-nomination. It could be combined with the development of short questionnaires that were tailored to the specific domain knowledge, or with the measures of personal characteristics that have been found recently to improve forecasting in other fields (Mellers et al, 2015). However, our results show that the development of a more structured mechanism for distinguishing expertise in demand forecasting judgmental adjustments is a task worth attempting.…”