Calibrating ensemble forecasting models with sparse data in the social sciences

Montgomery, Jacob M.; Hollenbach, Florian M.; Ward, Michael D.

doi:10.1016/j.ijforecast.2014.08.001

Cited by 17 publications

(6 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Prior applications of out-of-sample prediction to SAOMs include Desmarais and Cranmer (2012a), Kinne (2013), Koskinen and Edling (2012), and Warren (2016). There is also a substantial literature looking at out-of-sample prediction with other network models (e.g., Chiba et al, 2015;Montgomery et al, 2015;Ward et al, 2013).…”

Section: Out-of-sample Predictionmentioning

confidence: 99%

A theoretical and empirical comparison of the temporal exponential random graph model and the stochastic actor-oriented model

Leifeld¹,

Cranmer²

2019

Net Sci

View full text Add to dashboard Cite

The temporal exponential random graph model (TERGM) and the stochastic actor-oriented model (SAOM, e. g., SIENA) are popular models for longitudinal network analysis. We compare these models theoretically, via simulation, and through a real-data example in order to assess their relative strengths and weaknesses. Though we do not aim to make a general claim about either being superior to the other across all specifications, we highlight several theoretical differences the analyst might consider and find that with some specifications, the two models behave very similarly, while each model out-predicts the other one the more the specific assumptions of the respective model are met. * The authors wish to thank the many people who have provided extensive comments on this paper. All errors remain our own.

show abstract

Section: Out-of-sample Predictionmentioning

confidence: 99%

A theoretical and empirical comparison of the temporal exponential random graph model and the stochastic actor-oriented model

Leifeld¹,

Cranmer²

2019

Net Sci

View full text Add to dashboard Cite

show abstract

“…It has been shown to be useful in the predicting the outcomes of American football games (Lee, Danileiko, & Vi, 2018). Finally, many researchers (e.g., Baron, Mellers, Tetlock, Stone, & Ungar, 2014;Lee & Danileiko, 2014;Merkle, 2010;Montgomery, Hollenbach, & Ward, 2015;Raftery, Gneiting, Balabdaoui, & Polakowski, 2005;Satopää et al, 2014;Shlomi & Wallsten, 2010;Turner, Steyvers, Merkle, Budescu, & Wallsten, 2014) have discussed "recalibration" and related methods, whereby forecasts are corrected for systematic forecaster biases and distortions (though also see Kadane & Fischhoff, 2013). Many of these methods were developed and successfully applied in the Aggregative Contingent Estimation (ACE) forecasting tournament, sponsored by IARPA from about 2011-2015.…”

mentioning

confidence: 99%

Beating the average forecast: Regularization based on forecaster attributes

Merkle

Saw

Davis‐Stober

2020

Journal of Mathematical Psychology

View full text Add to dashboard Cite

In a variety of real-world forecasting contexts, researchers have demonstrated that the unweighted average forecast is reasonably accurate and difficult to improve upon with more complex, model-based aggregation methods. We investigate this phenomenon by systematically examining the relationship between individual forecaster characteristics (e.g., bias, consistency) and aspects of the criterion being forecast (e.g., "signal strength"). To this end, we develop a model inspired by Cultural Consensus Theory (Batchelder & Romney, 1988) that (i) allows us to jointly estimate both forecaster characteristics and environmental characteristics and (ii) contains the unweighted average as a special case. This allows us to use the model as a regularization method for forecast aggregation, where restrictions on forecaster parameters make the model similar to use of an unweighted average. Relatedly, the model allows us to apply existing results on optimal forecaster weighting to real data. We show how the model provides guidance for identifying prediction environments where the average forecast can potentially be beaten. We also conduct two simulation studies and illustrate the model's practical application using forecasts of Australian Football League point spreads. BackgroundA considerable body of work in forecasting has established that taking a simple, unweighted average of forecasters yields a forecast that is not only good, it is extremely difficult to improve upon with more complex aggregation methods (

show abstract

“…Draws onMontgomery, Hollenbach and Ward (2014).7 An example of a public domain ensemble site is at http://www.atmos.washington.edu/~ens/uwme. cgi.…”

mentioning

confidence: 99%

Irregular leadership changes in 2014: Forecasts using ensemble, split-population duration models

Beger

Dorff

Ward

2016

International Journal of Forecasting

Self Cite

View full text Add to dashboard Cite

We forecast Irregular Leadership Changes (ILC)-unexpected leadership changes in contravention of a state's established laws and conventions-for mid-2014 using predictions generated from an innovative ensemble model that is composed of several split-population duration regression models. This approach uses distinct thematic models, combining them into one aggregate forecast developed on the basis of their predictive accuracy and uniqueness. The data are based on 45 ILCs that occurred from

show abstract

Calibrating ensemble forecasting models with sparse data in the social sciences

Cited by 17 publications

References 67 publications

A theoretical and empirical comparison of the temporal exponential random graph model and the stochastic actor-oriented model

A theoretical and empirical comparison of the temporal exponential random graph model and the stochastic actor-oriented model

Beating the average forecast: Regularization based on forecaster attributes

Irregular leadership changes in 2014: Forecasts using ensemble, split-population duration models

Contact Info

Product

Resources

About