Replication of the natural selection of bad science

Kohrt, Florian; Smaldino, Paul E.; McElreath, Richard; Schönbrodt, Felix D.

doi:10.1098/rsos.221306

Cited by 6 publications

(4 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The model demonstrates that this way of conducting research, called "bad science", can rapidly spread through a group of laboratories, and become its dominant approach. The main results of the model were recently replicated [33], and its framework has been used to conduct virtual experiments of interventions such as auditing of research facilities, making publication of negative results more prestigious, improving peer review, assigning research funds randomly or according to methodological integrity, and researchers expending effort on the selection of strong hypotheses through theory development [34][35][36].…”

Section: Plos Onementioning

confidence: 99%

Impact of redefining statistical significance on P-hacking and false positive rates: An agent-based model

Fitzpatrick,

Gorman,

Trombatore

2024

PLoS ONE

View full text Add to dashboard Cite

In recent years, concern has grown about the inappropriate application and interpretation of P values, especially the use of P<0.05 to denote “statistical significance” and the practice of P-hacking to produce results below this threshold and selectively reporting these in publications. Such behavior is said to be a major contributor to the large number of false and non-reproducible discoveries found in academic journals. In response, it has been proposed that the threshold for statistical significance be changed from 0.05 to 0.005. The aim of the current study was to use an evolutionary agent-based model comprised of researchers who test hypotheses and strive to increase their publication rates in order to explore the impact of a 0.005 P value threshold on P-hacking and published false positive rates. Three scenarios were examined, one in which researchers tested a single hypothesis, one in which they tested multiple hypotheses using a P<0.05 threshold, and one in which they tested multiple hypotheses using a P<0.005 threshold. Effects sizes were varied across models and output assessed in terms of researcher effort, number of hypotheses tested and number of publications, and the published false positive rate. The results supported the view that a more stringent P value threshold can serve to reduce the rate of published false positive results. Researchers still engaged in P-hacking with the new threshold, but the effort they expended increased substantially and their overall productivity was reduced, resulting in a decline in the published false positive rate. Compared to other proposed interventions to improve the academic publishing system, changing the P value threshold has the advantage of being relatively easy to implement and could be monitored and enforced with minimal effort by journal editors and peer reviewers.

show abstract

Section: Plos Onementioning

confidence: 99%

Impact of redefining statistical significance on P-hacking and false positive rates: An agent-based model

Fitzpatrick,

Gorman,

Trombatore

2024

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…Although simulation studies are a powerful tool for methodological research, results from those studies, as the example of the 'one-in-ten rule' illustrates, are not definitive. Like empirical results, results from simulation studies need to be reproduced and replicated to verify their veracity [2], and this is increasingly called for [8][9][10]. So far, there is limited evidence on the reproducibility or replicability of simulation studies.…”

Section: Introductionmentioning

confidence: 99%

Replicability of simulation studies for the investigation of statistical methods: the RepliSims project

Luijken,

Lohmann,

Alter

et al. 2024

R. Soc. Open Sci.

View full text Add to dashboard Cite

Results of simulation studies evaluating the performance of statistical methods can have a major impact on the way empirical research is implemented. However, so far there is limited evidence of the replicability of simulation studies. Eight highly cited statistical simulation studies were selected, and their replicability was assessed by teams of replicators with formal training in quantitative methodology. The teams used information in the original publications to write simulation code with the aim of replicating the results. The primary outcome was to determine the feasibility of replicability based on reported information in the original publications and supplementary materials. Replicasility varied greatly: some original studies provided detailed information leading to almost perfect replication of results, whereas other studies did not provide enough information to implement any of the reported simulations. Factors facilitating replication included availability of code, detailed reporting or visualization of data-generating procedures and methods, and replicator expertise. Replicability of statistical simulation studies was mainly impeded by lack of information and sustainability of information sources. We encourage researchers publishing simulation studies to transparently report all relevant implementation details either in the research paper itself or in easily accessible supplementary material and to make their simulation code publicly available using permanent links.

show abstract

“…The paper's conclusions remain unaffected after correcting for these errors. For further discussion, see [1], which also presents a reimplementation of the model in the R programming language, archived in the Software Heritage archive at https://archive. softwareheritage.org/swh:1:snp:60ab9f391840fbb0d226fdbce35169b 271e00918;origin=https://gitlab.com/fkohrt/bachelorarbeit-code.…”

mentioning

confidence: 99%

mentioning

confidence: 99%

Correction to: ‘The natural selection of bad science’ (2016) by Paul E. Smaldino and Richard McElreath

Smaldino,

McElreath

2023

R. Soc. open sci.

Self Cite

View full text Add to dashboard Cite

Replication of the natural selection of bad science

Cited by 6 publications

References 27 publications

Impact of redefining statistical significance on P-hacking and false positive rates: An agent-based model

Impact of redefining statistical significance on P-hacking and false positive rates: An agent-based model

Replicability of simulation studies for the investigation of statistical methods: the RepliSims project

Correction to: ‘The natural selection of bad science’ (2016) by Paul E. Smaldino and Richard McElreath

Contact Info

Product

Resources

About