Quantifying machine influence over human forecasters

Abstract

Crowdsourcing human forecasts and machine learning models each show promise in predicting future geopolitical outcomes. Crowdsourcing increases accuracy by pooling knowledge, which mitigates individual errors. On the other hand, advances in machine learning have led to machine models that increase accuracy due to their ability to parameterize and adapt to changing environments. To capitalize on the unique advantages of each method, recent efforts have shown improvements by “hybridizing” forecasts—pairing human forecasters with machine models. This study analyzes the effectiveness of such a hybrid system. In a perfect world, independent reasoning by the forecasters combined with the analytic capabilities of the machine models should complement each other to arrive at an ultimately more accurate forecast. However, well-documented biases describe how humans often mistrust and under-utilize such models in their forecasts. In this work, we present a model that can be used to estimate the trust that humans assign to a machine. We use forecasts made in the absence of machine models as prior beliefs to quantify the weights placed on the models. Our model can be used to uncover other aspects of forecasters’ decision-making processes. We find that forecasters trust the model rarely, in a pattern that suggests they treat machine models similarly to expert advisors, but only the best forecasters trust the models when they can be expected to perform well. We also find that forecasters tend to choose models that conform to their prior beliefs as opposed to anchoring on the model forecast. Our results suggest machine models can improve the judgment of a human pool but highlight the importance of accounting for trust and cognitive biases involved in the human judgment process.

Deepfake detection by human crowds, machines, and machine-informed crowds

“…Moreover, the model’s incorrect assessment of the political leader deepfake videos is associated with a decrease in participant accuracy, which is in line with recent empirical research that shows deepfake warnings do not improve discernment of political videos ( 71 ). Likewise, these results mirror other recent research revealing human–AI collaborative decision-making does not necessarily lead to more accurate results than either humans or AI alone ( 72 – 76 ).…”

Hybrid forecasting of geopolitical events†

“…While this is evidence of their value, it provides further evidence that forecasters must have enough expertise to know when and how to use this information (Abeliuk et al. 2020). Predicting the future is difficult, especially for deeply uncertain, impactful geopolitical events.…”

Models, Markets, and the Forecasting of Elections

“…Along similar lines, one could include prediction market prices as inputs into existing models, along with trial-heat polls and fundamental factors, calibrated based on historical data. Alternatively, one could make model outputs available to human forecasters to use as they see fit when soliciting their beliefs about events (Abeliuk et al, 2020). Yet another possibility involves tapping the wisdom of crowds by asking survey respondents to predict the voting behavior of those in their social circles (Galesic et al, 2018).…”