What do forecasting rationales reveal about thinking patterns of top geopolitical forecasters?

Karvetski, Christopher W.; Meinel, Carolyn; Maxwell, Daniel T.; Lu, Yunzi; Mellers, Barbara A.

doi:10.1016/j.ijforecast.2021.09.003

Cited by 15 publications

(5 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Delta Brier Score is based on the difference between the Brier score of the consensus estimate on a given question at a given time, and a forecaster's individual estimate for this question at this time. This version was used in Karvetski et al (2021). It is reverse-scored in the current analysis, so that higher values denote worse accuracy, consistent with other outcome measures in this chapter.…”

Section: Outcome Variablesmentioning

confidence: 81%

“…Similarly, a positive correlation was observed between integratively complex thought protocols and calibration (r = 0.31). Karvetski et al (2021) documented the relationship between linguistic properties of forecast rationales and accuracy in more recent forecasting tournaments, including ACE and the global forecasting challenge. In their study, the outcome is a Delta-Brier measure where higher scores denote better accuracy.…”

Section: Rationale Text Featuresmentioning

confidence: 99%

See 1 more Smart Citation

Talent Spotting in Crowd Prediction

Atanasov¹,

Himmelstein

2023

International Series in Operations Research &Amp; Management Science

View full text Add to dashboard Cite

Who is good at prediction? Addressing this question is key to recruiting and cultivating accurate crowds and effectively aggregating their judgments. Recent research on superforecasting has demonstrated the importance of individual, persistent skill in crowd prediction. This chapter takes stock of skill identification measures in probability estimation tasks, and complements the review with original analyses, comparing such measures directly within the same dataset. We classify all measures in five broad categories: 1) accuracy-related measures, such as proper scores, modelbased estimates of accuracy and excess volatility scores; 2) intersubjective measures, including proxy, surrogate and similarity scores; 3) forecasting behaviors, including activity, belief updating, extremity, coherence, and linguistic properties of rationales; 4) dispositional measures of fluid intelligence, cognitive reflection, numeracy, personality and thinking styles; and 5) measures of expertise, including demonstrated knowledge, confidence calibration, biographical, and self-rated expertise. Among non-accuracy-related measures, we report a median correlation coefficient with outcomes of r = 0.20. In the absence of accuracy data, we find that intersubjective and behavioral measures are most strongly correlated with forecasting accuracy. These results hold in a LASSO machine-learning model with automated variable selection. Two focal applications provide context for these assessments: long-term prediction and corporate forecasting tournaments.

show abstract

Section: Outcome Variablesmentioning

confidence: 81%

Section: Rationale Text Featuresmentioning

confidence: 99%

Talent Spotting in Crowd Prediction

Atanasov¹,

Himmelstein

2023

International Series in Operations Research &Amp; Management Science

View full text Add to dashboard Cite

show abstract

“…They also were better than individuals who could gather social information but not discuss. As an additional confirmation, individual characteristics facilitating deliberation with others, such as cognitive flexibility and open-mindedness [ 8 ], cooperativeness within teams [ 9 ] as well as the tendency to provide more articulate rationales [ 10 ] could distinguish the best forecasters in the competition.…”

Section: Introductionmentioning

confidence: 99%

Compromising improves forecasting

2023

View full text Add to dashboard Cite

Predicting the future can bring enormous advantages. Across the ages, reliance on supernatural foreseeing was substituted by the opinion of expert forecasters, and now by collective intelligence approaches which draw on many non-expert forecasters. Yet all of these approaches continue to see individual forecasts as the key unit on which accuracy is determined. Here, we hypothesize that compromise forecasts, defined as the average prediction in a group, represent a better way to harness collective predictive intelligence. We test this by analysing 5 years of data from the Good Judgement Project and comparing the accuracy of individual versus compromise forecasts. Furthermore, given that an accurate forecast is only useful if timely, we analyze how the accuracy changes through time as the events approach. We found that compromise forecasts are more accurate, and that this advantage persists through time, though accuracy varies. Contrary to what was expected (i.e. a monotonous increase in forecasting accuracy as time passes), forecasting error for individuals and for team compromise starts its decline around two months prior to the event. Overall, we offer a method of aggregating forecasts to improve accuracy, which can be straightforwardly applied in noisy real-world settings.

show abstract

“…distributional information, is recommended by Armstrong (2005) and is part of professional forecasters and analysts' training (Tetlock & Gardner, 2016), which is shown to improve their performance (Chang et al, 2016). Especially, Karvetski et al (2021) show that the use of base rates has a positive effect on forecast accuracy but in general there has been paid more attention to the biases than to debiasing (Chang et al, 2016). Green and Armstrong (2007) describe a procedure to include analogies in the forecasting process and Lovallo et al (2012) conduct an empirical study using the outside view to forecast stock returns but both suffer from a subjective choice of similar objects such that the resulting reference classes are prone to the availability bias described by Tversky and Kahneman (1973).…”

Section: Introductionmentioning

confidence: 99%

“…Reference classes and outside views are established in forecasting literature and practice, e.g., Armstrong (2005) and (Tetlock and Gardner, 2016) recommend the use of base rates, i.e. distributional information, known to improve forecasting performances (Chang et al, 2016;Karvetski et al, 2021). But in general the literature focused more on biases than on debiasing itself (Chang et al, 2016).…”

Section: Introductionmentioning

confidence: 99%

Reference class selection in similarity‐based forecasting of corporate sales growth

Theising

Wied

Ziggel

2022

Journal of Forecasting

View full text Add to dashboard Cite

This paper proposes a general method to handle forecasts exposed to behavioral bias by finding appropriate outside views, in our case corporate sales forecasts of analysts. The idea is to find reference classes, that is, peer groups, for each analyzed company separately that share similarities to the firm of interest with respect to a specific predictor. The classes are regarded to be optimal if the forecasted sales distributions match the actual distributions as closely as possible. The forecast quality is measured by applying goodness‐of‐fit tests on the estimated probability integral transformations and by comparing the predicted quantiles. The method is out‐of‐sample backtested on a data set consisting of 21,808 US firms over the time period 1950–2019, which is also descriptively analyzed. It appears that, in particular, the past operating margins are good predictors for the distribution of future sales. A case study compares the outside view of our distributional forecasts with actual analysts' forecasts and emphasizes the relevance of our approach in practice.

show abstract

What do forecasting rationales reveal about thinking patterns of top geopolitical forecasters?

Cited by 15 publications

References 41 publications

Talent Spotting in Crowd Prediction

Talent Spotting in Crowd Prediction

Compromising improves forecasting

Reference class selection in similarity‐based forecasting of corporate sales growth

Contact Info

Product

Resources

About