Insights into the accuracy of social scientists’ forecasts of societal change

Grossmann, Igor; Rotella, Amanda; Hutcherson, Cendri A.; Sharpinskyi, Konstantyn; Varnum, Michael E. W.; Achter, Sebastian; Dhami, Mandeep K.; Guo, Xinqi Evie; Kara-Yakoubian, Mane; Mandel, David R.; Raes, Louis; Tay, Louis; Vié, Aymeric; Wagner, Lisa; Adamkovič﻿﻿﻿, Matúš; Arami, Arash; Arriaga, Patrí­cia; Bandara, Kasun; Baník, Gabriel; Bartoš, František; Baskin, Ernest; Bergmeir, Christoph; Białek, Michał; Børsting, Caroline K.; Browne, Dillon T.; Caruso, Eugene M.; Chen, Rong; Chie, Bin‐Tzong; Chopik, William J.; Collins, Robert N.; Cong, Chin Wen; Conway, Lucian Gideon; Davis, Matthew; Day, Martin V.; Dhaliwal, Nathan A.; Durham, Justin D.; Dziekan, Martyna; Elbæk, Christian T.; Shuman, Eric; Fabrykant, Marharyta; Firat, Mustafa; Fong, Geoffrey T.; Frimer, Jeremy A.; Gallegos, Jonathan; Goldberg, Simon B.; Gollwitzer, Anton; Goyal, Julia; Graf‐Vlachy, Lorenz; Gronlund, Scott D.; Hafenbrädl, Sebastian; Hartanto, Andree; Hirshberg, Matthew J.; Hornsey, Matthew J.; Howe, Piers D. L.; Izadi, Anoosha; Jaeger, Bastian; Kačmár, Pavol; Kim, Yeun Joon; Krenzler, Ruslan; Lannin, Daniel G.; Lin, Haitao; Lou, Nigel Mantou; Lua, Verity Y. Q.; Lukaszewski, Aaron W.; Ly, Albert L.; Madan, Christopher R.; Maier, Maximilian; Majeed, Nadyanna M.; March, David S.; Marsh, Abigail A.; Misiak, Michał; Myrseth, Kristian Ove R.; Napan, Jaime M.; Nicholas, Jonathan; Νικολόπουλος, Κωνσταντίνος; Jiaqing, O; Otterbring, Tobias; Paruzel‐Czachura, Mariola; Pauer, Shiva; Protzko, John; Raffaelli, Quentin; Ropovik, Ivan; Ross, Robert M.; Roth, Yefim; Røysamb, Espen; Schnabel, Landon; Schütz, Astrid; Seifert, Matthias; Sevincer, A. Timur; Sherman, Garrick; Simonsson, Otto; Sung, Ming‐Chien; Tai, Chung-Ching; Talhelm, Thomas; Teachman, Bethany A.; Tetlock, Philip E.; Thomakos, Dimitrios D.; Tse, Dwight C. K.; Twardus, Oliver; Tybur, Joshua M.; Ungar, Lyle; Vandermeulen, Daan; Williams, Leighton Vaughan; Vosgerichian, Hrag A.; Wang, Qi; Wang, Ke; Whiting, Mark E.; Wollbrant, Conny; Yang, Tao; Yogeeswaran, Kumar; Yoon, Sangsuk; Alves, Ventura R; Andrews‐Hanna, Jessica R.; Bloom, Paul Alexander; Boyles, Anthony; Charis, Loo; Choi, Mingyeong; Darling-Hammond, Sean; Ferguson, Z. E.; Kaiser, Cheryl R.; Karg, Simon Tobias; Ortega, Alberto López; Mahoney, Lori; Marsh, Melvin S.; Martinie, Marcellin; Michaels, Eli K.; Millroth, Philip; Naqvi, Jeanean B.; Ng, Weiting; Rutledge, Robb B.; Slattery, Peter; Smiley, Adam Herschel; Strijbis, Oliver; Sznycer, Daniel; Tsukayama, Eli; Loon, Austin van; Voelkel, Jan G.; Wienk, Margaux N. A.; Wilkening, Tom

doi:10.1038/s41562-022-01517-1

Cited by 37 publications

(14 citation statements)

References 65 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When the forecasts were compared to the true outcomes, Grossmann et al discovered that the forecasts of the social scientists were not particularly impressive. They conclude that "social scientists' forecasts were on average no more accurate than simple statistical models" 5 , a finding that is consistent with previous research 2 . Grossmann et al 5 also found that "for most domains, social scientists' predictions were either similar to or worse than the [nonexpert] crowd's prediction".…”

supporting

confidence: 85%

“…They conclude that "social scientists' forecasts were on average no more accurate than simple statistical models" 5 , a finding that is consistent with previous research 2 . Grossmann et al 5 also found that "for most domains, social scientists' predictions were either similar to or worse than the [nonexpert] crowd's prediction". However, these findings are also consistent with a slightly different comparison that makes the social scientists look a bit better: in five domains, social scientists beat the nonexpert crowd; in one domain, the nonexpert crowd beat the social scientists; and in six domains, it was not possible to declare a clear winner.…”

supporting

confidence: 85%

“…For example, researchers have found that intelligence analysts show some skill in making certain kinds of social forecasts 3,4 . Against this background of previous work, the paper by Grossmann et al 5 adds to our understanding of expert forecasting in social systems and raises some important new questions.…”

mentioning

confidence: 96%

See 2 more Smart Citations

Predicting the future of society

Salganik

2023

Nat Hum Behav

View full text Add to dashboard Cite

supporting

confidence: 85%

supporting

confidence: 85%

See 1 more Smart Citation

Predicting the future of society

Salganik

2023

Nat Hum Behav

View full text Add to dashboard Cite

“…Although our conclusions are limited by the small number of specific domains assessed here, they nevertheless suggest that scientists may use their greater knowledge of research and theory to justify, rather than shape, the intuitions they share with the average person 3 . This work, along with other forecasting tournaments among social scientists (The Forecasting et al, 2023), raises important questions about how to improve the accuracy of scientists’ predictions regarding the societal effects of events like the COVID-19 pandemic.…”

Section: Discussionmentioning

confidence: 99%

On the accuracy, media representation, and public perception of psychological scientists’ judgments of societal change.

Hutcherson¹,

Sharpinskyi²,

Varnum³

et al. 2023

American Psychologist

Self Cite

View full text Add to dashboard Cite

At the onset of the COVID-19 pandemic, psychological scientists frequently made on-the-record predictions in public media about how individuals and society would change. Such predictions were often made outside these scientists’ areas of expertise, with justifications based on intuition, heuristics, and analogical reasoning (Study 1; N = 719 statements). How accurate are these kinds of judgments regarding societal change? In Study 2, we obtained predictions from scientists (N = 717) and lay Americans (N = 394) in Spring 2020 regarding the direction of change for a range of social and psychological phenomena. We compared them to objective data obtained at 6 months and 1 year. To further probe how experience impacts such judgments, 6 months later (Study 3), we obtained retrospective judgments of societal change for the same domains (Nscientists = 270; Nlaypeople = 411). Bayesian analysis suggested greater credibility of the null hypothesis that scientists’ judgments were at chance on average for both prospective and retrospective judgments. Moreover, neither domain-general expertise (i.e., judgmental accuracy of scientists compared to laypeople) nor self-identified domain-specific expertise improved accuracy. In a follow-up study on meta-accuracy (Study 4), we show that the public nevertheless expects psychological scientists to make more accurate predictions about individual and societal change compared to most other scientific disciplines, politicians, and nonscientists, and they prefer to follow their recommendations. These findings raise questions about the role psychological scientists could and should play in helping the public and policymakers plan for future events.

show abstract

“…These innovations include fine-grained data (1), novel methods and algorithms (2), systematic expert surveys (3) and the usage of alternative early warning proxies such as news reports (4) or financial market developments (5,6). In addition, the prediction community has become increasingly aware of potential forecasting biases and the benefits of simple data-driven models (7).…”

mentioning

confidence: 99%

Counterfactual coercion: Could harsher sanctions against Russia have prevented the worst?

Niemeier

Schneider

2023

Preprint

View full text Add to dashboard Cite

Recent studies show that properly designed economic sanctions can force the target to refrain from violating international norms. However, policy-makers are not enabled to integrate this finding in their ex ante assessments whether more intensive coercive measures could prevent military coups, human rights violations, or a war of aggression such as the Russian invasion of Ukraine. In this article, we address this shortcoming of the conflict forecasting literature and introduce counterfactual predictions to answer the what if-question of whether adequate sanctions by the European Union and the United States could have provoked targets to abandon severe norms violations. To this end, a training data set for the period from 1989 to 2008 is used to predict the incidence, intensity, and success of sanctions from 2009 to 2015. Our policy counterfactuals for key sanction cases demonstrate that stricter EU coercion against Russia after the annexation of Crimea could have triggered policy concessions from the regime of President Putin.

show abstract

Insights into the accuracy of social scientists’ forecasts of societal change

Cited by 37 publications

References 65 publications

Predicting the future of society

Predicting the future of society

On the accuracy, media representation, and public perception of psychological scientists’ judgments of societal change.

Counterfactual coercion: Could harsher sanctions against Russia have prevented the worst?

Contact Info

Product

Resources

About