Evaluating replicability of laboratory experiments in economics

Camerer, Colin F.; Dreber, Anna; Forsell, Eskil; Ho, Teck-Hua; Huber, Jürgen; Johannesson, Magnus; Kirchler, Michael; Almenberg, Johan; Altmejd, Adam; Chan, Taizan; Heikensten, Emma; Holzmeister, Felix; Imai, Taisuke; Isaksson, Siri; Nave, Gideon; Pfeiffer, Thomas; Razen, Michael; Wu, Hang

doi:10.1126/science.aaf0918

Cited by 1,083 publications

(955 citation statements)

References 60 publications

Supporting

Mentioning

856

Contrasting

Unclassified

Order By: Relevance

“…This issue has been highlighted by recent large replication attempts in psychology and experimental economics (Open Science Collaboration, 2015;Camerer et al, 2016) as well as by recent critical reviews of oxytocin research on humans (Nave et al, 2015;Lane et al, 2016).…”

Section: Introductionmentioning

confidence: 99%

Testosterone administration does not affect men's rejections of low ultimatum game offers or aggressive mood

Cueva

Roberts

Spencer

et al. 2017

Hormones and Behavior

View full text Add to dashboard Cite

Correlative evidence suggests that testosterone promotes dominance and aggression. However, causal evidence is scarce and offers mixed results. To investigate this relationship, we administered testosterone for 48h to 41 healthy young adult men in a within-subjects, double-blind placebo-controlled balanced crossover design. Subjects played the role of responders in an ultimatum game, where rejecting a low offer is costly, but serves to destroy the proposer's profit. Such action can hence be interpreted as non-physical aggression in response to social provocation. In addition, subjects completed a self-assessed mood questionnaire. As expected, self-reported aggressiveness was a key predictor of ultimatum game rejections. However, while testosterone affected subjective ratings of feeling energetic and interested, our evidence strongly suggests that testosterone had no effect on ultimatum game rejections or on aggressive mood. Our findings illustrate the importance of using causal interventions to assess correlative evidence.

show abstract

Section: Introductionmentioning

confidence: 99%

Testosterone administration does not affect men's rejections of low ultimatum game offers or aggressive mood

Cueva

Roberts

Spencer

et al. 2017

Hormones and Behavior

View full text Add to dashboard Cite

show abstract

“…A recent coordinated replication of 18 between-subject laboratory experiments from two top economics journals found that only 11 (61%) studies replicated as measured by p < .05, despite being powered at over 90% and sending study plans to original authors for verification (Camerer et al, 2016). Results for reproducing the results of nonexperimental studies using the same data and analyses 10 have been even worse, with rates below 50% (Chang & Li, 2015, 49% with original authors' assistance, 33% without; Dewald, Thursby, & Anderson, 1986, 13%;McCullough, McGeary, & Harrison, 2006, 23%).…”

Section: A Science-wide Problemmentioning

confidence: 99%

Open Science: What, Why, and How

Spellman¹,

Gilbert²,

Corker³

2017

Preprint

View full text Add to dashboard Cite

Open science is a collection of actions designed to make scientific processes more transparent and results more accessible. Its goal is to build a more replicable and robust science; it does so using new technologies, altering incentives, and changing attitudes. The current movement toward open science was spurred, in part, by a recent series of unfortunate events within psychology and other sciences. These events include the large number of studies that have failed to replicate and the prevalence of common research and publication procedures that could explain why. Many journals and funding agencies now encourage, require, or reward some open science practices, including pre-registration, providing full materials, posting data, distinguishing between exploratory and confirmatory analyses, and running replication studies. Individuals can practice and promote open science in their many roles as researchers, authors, reviewers, editors, teachers, and members of hiring, tenure, promotion, and awards committees. A plethora of resources are available to help scientists, and science, achieve these goals. Keywords: data sharing, file drawer problem, open access, open science, preregistration, questionable research practices, replication crisis, reproducibility, scientific integrityThanks to Brent Donnellan (big thanks!), Daniël Lakens, Calvin Lai, Courtney Soderberg, and Simine Vazire OPEN SCIENCE 3 Open Science When we (the authors) look back a couple of years, to the earliest outline of this chapter, the open science movement within psychology seemed to be in its infancy. Plenty of people were pointing to problems in psychology research, collecting archival data to support the claims, and suggesting how science could be improved. Now it seems that the open science movement has reached adolescence. Things are happening-and they are happening quickly. New professional organizations are being formed to uncover and facilitate ways to improve science, often through new technology, and some old organizations are adopting new procedures to remedy problems created by past practices, often involving revising journal policies. Researchers are changing the way they teach, practice, and convey science. And scientific information (and opinions) are traveling fast. In blogs, tweets, Facebook groups, op eds, science journalism, circulation of preprints, post-print comments, video talks, and so on, more people are engaged in communicating science, and hoping to improve science, than ever before. Thus, any new technology, new procedure, new website, or new controversy we describe is likely to be superseded (or solved) even by the time this chapter is published. But the core values of open science should remain. The "Open Science" MovementScience is about evidence: observing, measuring, collecting, and analyzing evidence. And it is about evidence that can be shared across observers and, typically, although not necessarily exactly (Merton, 1973;Popper, 1959), replicated later. Science is about testing hypotheses, using inductiv...

show abstract

“…When Dreber's team repeated 18 economics experiments as part of a follow-up to her psychology investigation, both the prediction markets and surveys of individuals overestimated the odds of each study's reproducibility 9 . Dreber isn't sure why this happened.…”

Section: Field-testing the Futurementioning

confidence: 99%

The power of prediction markets

Mann¹

2016

Nature

View full text Add to dashboard Cite

Evaluating replicability of laboratory experiments in economics

Cited by 1,083 publications

References 60 publications

Testosterone administration does not affect men's rejections of low ultimatum game offers or aggressive mood

Testosterone administration does not affect men's rejections of low ultimatum game offers or aggressive mood

Open Science: What, Why, and How

The power of prediction markets

Contact Info

Product

Resources

About