A survey of human judgement and quantitative forecasting methods

Abstract

This paper's top-level goal is to provide an overview of research conducted in the many academic domains concerned with forecasting. By providing a summary encompassing these domains, this survey connects them, establishing a common ground for future discussions. To this end, we survey literature on human judgement and quantitative forecasting as well as hybrid methods that involve both humans and algorithmic approaches. The survey starts with key search terms that identified more than 280 publications in the fields of computer science, operations research, risk analysis, decision science, psychology and forecasting. Results show an almost 10-fold increase in the application-focused forecasting literature between the 1990s and the current decade, with a clear rise of quantitative, data-driven forecasting models. Comparative studies of quantitative methods and human judgement show that (1) neither method is universally superior, and (2) the better method varies as a function of factors such as availability, quality, extent and format of data, suggesting that (3) the two approaches can complement each other to yield more accurate and resilient models. We also identify four research thrusts in the human/machine-forecasting literature: (i) the choice of the appropriate quantitative model, (ii) the nature of the interaction between quantitative models and human judgement, (iii) the training and incentivization of human forecasters, and (iv) the combination of multiple forecasts (both algorithmic and human) into one. This review surveys current research in all four areas and argues that future research in the field of human/machine forecasting needs to consider all of them when investigating predictive performance. We also address some of the ethical dilemmas that might arise due to the combination of quantitative models with human judgement.

The simple observational critical care studies: estimations by students, nurses, and physicians of in-hospital and 6-month mortality

“…The moderate accuracy of clinical predictions by individuals has been well established [ 24 ]. It is assumed that many supportive technical innovations in clinical decision-making will exponentially become available [ 25 ]. The challenge is to find and especially use the best possible algorithm to predict patient outcomes [ 26 ].…”

Speed-vs-Accuracy Tradeoff in Collective Estimation: An Adaptive Exploration-Exploitation Case

“…The aim of the exploration phase is to aggregate or to combine the information distributed within the collective [30]. The aggregation can be defined in two distinct domains, the information space and the spatial (physical) domain.…”

The Wisdom of the Coherent: Improving Correspondence with Coherence-Weighted Aggregation

“…From a methodological perspective, a focus on ensembles entails a greater degree of complexity in experimental variables. We need to better understand how the pillars of optimizationinformation representation, judgment elicitation, and post-judgment recalibration and aggregation-interact amongst themselves and how those interactions are moderated by task characteristics (Zellner et al, 2021).…”

“…One aim of the present work is to compare alternative coherence-weighted aggregation methods used in previous studies (e.g., Fan et al, 2019;Karvetski et al, 2013;Karvetski et al, 2020;Mandel et al, 2018;Predd et al, 2008;Wang et al, 2011) across multiple tasks and experiments. We wanted to determine whether the best coherence-weighting method-as determined by empirical accuracy-is stable or task dependent (Zellner et al, 2021). A second aim is to analyze whether individuals' out-of-sample coherence predicts correspondence; namely, whether coherence on one or more tasks predicted accuracy on another task.…”