Small Steps to Accuracy: Incremental Belief Updaters Are Better Forecasters

Atanasov, Pavel; Witkowski, Jens; Ungar, Lyle H.; Mellers, Barbara A.

doi:10.1145/3391403.3399540

Cited by 2 publications

(15 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Belief updating measures describe how forecasters update their forecasts over time. Atanasov et al (2020) distinguish between three measures: frequency (how often updates occur per question), magnitude (how large is the average update in absolute terms) and confirmation propensity (how often forecaster re-enter their most recent forecast). Rationale-based measures rely on text analysis of the rationales that forecasters write on the platform.…”

Section: Five Categories Of Skill Correlatesmentioning

confidence: 99%

“…We do not provide a formal meta-analysis, mainly because the wide range of research designs makes such estimates tricky to aggregate or compare. Study 2 aims to address this comparability issue and provide more in-depth coverage: we reconstruct a subset of measures across each category and assess their correlations with accuracy, both in-sample and out-of-sample, using the same data and a uniform analytical framework originally developed and described in Atanasov et al (2020).…”

Section: Five Categories Of Skill Correlatesmentioning

confidence: 99%

“…Calibration and Discrimination measures are facets of Brier score decomposition originally defined by Murphy & Winkler (1987). We use the operationalization in the context of individual forecasters in GJP, as described in Atanasov et al (2020).…”

Section: Accuracy-relatedmentioning

confidence: 99%

“…Activity: Such measures assess forecaster task engagement and vary based on the specific features available in a forecasting platform. Activities measured on the GJP platforms included: the total number of forecasts submitted over the course of a season, the number of unique questions a forecaster answers (by reporting a probability estimate), the number of unique sessions on the forecasting platform, the per-session average or the total time spent on the forecasting platform, and the number of clicks on news articles served by the platform (Mellers et al, 2015a, Atanasov et al, 2020.…”

Section: Behavioralmentioning

confidence: 99%

“…Second, we provide a quantitative summary of effect sizes in pre-existing studies, expressed as correlation coefficients between predictor measures and performance outcome measures. Third, we address measurement challenges inherent to cross-study comparisons by conducting a new analysis of GJP data in which we recreate a subset of the predictor measures across the five categories, following the analytical framework originally developed in Atanasov, Witkowski et al (2020, henceforth Atanasov et al, 2020. This chapter does not aim to develop a unified theory of what makes a great forecaster.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Talent Spotting in Crowd Prediction

Atanasov¹,

Himmelstein²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Who is good at prediction? Addressing this question is key to recruiting and cultivating accurate crowds and effectively aggregating their judgments. Recent research on superforecasting has demonstrated the importance of individual, persistent skill in crowd prediction. This chapter takes stock of skill identification measures in probability estimation tasks, and complements the review with original analyses, comparing such measures directly within the same dataset. We classify all measures in five broad categories: 1) accuracy-related measures, such as proper scores, model-based estimates of accuracy and excess volatility scores; 2) intersubjective measures, including proxy, surrogate and similarity scores; 3) forecasting behaviors, including activity, belief updating, extremity, coherence, and linguistic properties of rationales; 4) dispositional measures of fluid intelligence, cognitive reflection, numeracy, personality and thinking styles; and 5) measures of expertise, including demonstrated knowledge, confidence calibration, biographical, and self-rated expertise. Among non-accuracy-related measures, we report a median correlation coefficient with outcomes of r = 0.20. In the absence of accuracy data, we find that intersubjective and behavioral measures are most strongly correlated with forecasting accuracy. These results hold in a LASSO machine-learning model with automated variable selection. Two focal applications provide context for these assessments: long-term, existential risk prediction and corporate forecasting tournaments.

show abstract

Section: Five Categories Of Skill Correlatesmentioning

confidence: 99%

Section: Five Categories Of Skill Correlatesmentioning

confidence: 99%

Section: Accuracy-relatedmentioning

confidence: 99%

Section: Behavioralmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Talent Spotting in Crowd Prediction

Atanasov¹,

Himmelstein²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

It depends coaching – The most fundamental, simple and complex principle or a mere copout?

Collins

Taylor

Ashford

et al. 2022

Sports Coaching Review

View full text Add to dashboard Cite

A growing body of literature identifies the importance of conditionality in coaching and coach education. In simple terms, the complexity and hyperdynamic nature of many coaching environments makes it increasingly unlikely that any one approach, tool or even paradigm will be appropriate to every situation. Accordingly, in this paper, we critically consider the use of Professional judgment and Decision-Making (often referred to as "it depends" coaching) as a valid approach to coaching, using three exemplar constructs backed by both literature and practical challenges. We conclude by comparing the underpinnings and logic of this approach and presenting implications for coaching practice. Our intention is to stimulate debate amongst coaches, coach developers and researchers.

show abstract

Small Steps to Accuracy: Incremental Belief Updaters Are Better Forecasters

Cited by 2 publications

References 1 publication

Talent Spotting in Crowd Prediction

Talent Spotting in Crowd Prediction

It depends coaching – The most fundamental, simple and complex principle or a mere copout?

Contact Info

Product

Resources

About