Forecasting forecaster accuracy: Contributions of past performance and individual differences

Abstract

A growing body of research indicates that forecasting skill is a unique and stable trait: forecasters with a track record of high accuracy tend to maintain this record. But how does one identify skilled forecasters effectively? We address this question using data collected during two seasons of a longitudinal geopolitical forecasting tournament. Our first analysis, which compares psychometric traits assessed prior to forecasting, indicates intelligence consistently predicts accuracy. Next, using methods adapted from classical test theory and item response theory, we model latent forecasting skill based on the forecasters’ past accuracy, while accounting for the timing of their forecasts relative to question resolution. Our results suggest these methods perform better at assessing forecasting skill than simpler methods employed by many previous studies. By parsing the data at different time points during the competitions, we assess the relative importance of each information source over time. When past performance information is limited, psychometric traits are useful predictors of future performance, but, as more information becomes available, past performance becomes the stronger predictor of future accuracy. Finally, we demonstrate the predictive validity of these results on out-of-sample data, and their utility in producing performance weights for wisdom-of-crowds aggregations.

The Wisdom of Timely Crowds

“…forecast quality (Atanasov et al, 2017;Budescu & Chen, 2014;Himmelstein et al, 2021). This raises an additional question: If larger crowds produce more accurate forecasts, can one just use large crowds to offset the difficulty in forecasting questions that are very far from their resolution date?…”

“…For example, because Brier scores are bounded between 0 and 2, linear models that use them as response variables cannot have normally distributed residuals. Therefore, Himmelstein et al (2021) transformed Brier Scores onto a scale that is more in line with this assumption. First, Brier Scores were transformed to a unit interval, into a metric that could be thought of as simply accuracy, as…”

“…Because accuracy is on a unit interval, with higher values representing more accurate forecasts, it can be transformed using traditional linking methods, such as the probit or logistic functions. Both Himmelstein et al (2021) and Merkle et al (2016) opted for the latter, as it transforms the accuracy measure onto a standard normal distribution, facilitating interpretability.…”

“…Although accounting for the timing of forecasts can help discriminate the skill of individual forecasters, Himmelstein et al, (2021) found it was not necessary for optimally weighting different forecasters. Using simple standardized accuracy measures as performance weights, which did not account for forecast timing, produced aggregate forecasts that were very similar in terms of accuracy to the more sophisticated model-based methods described here.…”

Recalibrating probabilistic forecasts to improve their accuracy

“…This choice is motivated and justified by the fact that accuracy is, typically, the top desideratum of good forecasts, and that the Brier Scores are considered by many the "gold standard". For example, they are often used in forecasting competitions (e.g., Himmelstein, Atanasov & Budescu, 2021;Mellers et al, 2014). However, as some of the reviewers of this manuscript have pointed out, this is not the sole criterion one could consider and, in fact, several appealing alternatives are well documented (e.g., .…”