The logical structure of experiments lays the foundation for a theory of reproducibility

Buzbas, Erkan O.; Devezer, Berna; Baumgaertner, Bert

doi:10.1098/rsos.221042

Cited by 10 publications

(3 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…After all, future research may reveal that current replication rates are "acceptable" (Bird, 2020;Freiling et al, 2021, p. 692;Guttinger, 2020, p. 8;Lewandowsky & Oberauer, 2020). Alternatively, the meaningfulness of quantifying replication rates may be called into question (Buzbas et al, 2023;Rubin, 2021a).…”

Section: Unspecified Replication Rate Targetsmentioning

confidence: 99%

Questionable Metascience Practices

Rubin

2023

JOTE

View full text Add to dashboard Cite

Questionable research practices may reduce the public’s trust in science. The present article considers some questionable metascience practices (QMPs) that may threaten scientists’ trust in metascience. A QMP is a research practice, assumption, or perspective that has been questioned by several commentators as being potentially problematic for the credibility of metascience and/or the science reform movement. The present article reviews 10 QMPs that relate to criticism, replication, bias, generalization, and the characterization of science. Specifically, the following QMPs are considered: (1) rejecting or ignoring self-criticism; (2) a fast ‘n’ bropen scientific criticism style; (3) overplaying the role of replication in science; (4) assuming a replication rate is “too low” without specifying an “acceptable” rate; (5) an unacknowledged metabias towards explaining the replication crisis in terms of researcher bias; (6) assuming that researcher bias can be reduced; (7) devaluing exploratory results as more “tentative” than confirmatory results; (8) presuming that QRPs are problematic research practices; (9) focusing on knowledge accumulation as an index of scientific progress; and (10) focusing on specific scientific methods. It is stressed that only some metascientists engage in some QMPs some of the time, and that these QMPs may not always be problematic. Research is required to estimate the prevalence and impact of QMPs. In the meantime, QMPs should be viewed as invitations to ask “questions” about how we go about doing metascience rather than as grounds for mistrusting the credibility of metascience.

show abstract

Section: Unspecified Replication Rate Targetsmentioning

confidence: 99%

Questionable Metascience Practices

Rubin

2023

JOTE

View full text Add to dashboard Cite

show abstract

“…Researchers can rely on ψ to gauge the ability of a given experimental design to elicit a target phenomenon with sufficient accuracy, which in some cases can lead to the development of alternative experimental approaches. We do emphasize that notions of effect size are just one of many factors that impact experimental outcomes; see Buzbas et al (2023) for a formal treatment of experimental design and its relation to replication rates.…”

Section: Recommendationsmentioning

confidence: 99%

Better Accuracy for Better Science . . . Through Random Conclusions

Davis‐Stober

Dana

Kellen

et al. 2023

Perspect Psychol Sci

View full text Add to dashboard Cite

Conducting research with human subjects can be difficult because of limited sample sizes and small empirical effects. We demonstrate that this problem can yield patterns of results that are practically indistinguishable from flipping a coin to determine the direction of treatment effects. We use this idea of random conclusions to establish a baseline for interpreting effect-size estimates, in turn producing more stringent thresholds for hypothesis testing and for statistical-power calculations. An examination of recent meta-analyses in psychology, neuroscience, and medicine confirms that, even if all considered effects are real, results involving small effects are indeed indistinguishable from random conclusions.

show abstract

“…Replication studies benefit to a large extent from open data and materials. However, it should be noted that open data and materials is not a prerequisite for replicating studies ( Buzbas, Devezer, & Baumgaertner, 2023 ). If highly sensitive data present challenges to open data principles, restricted access to data, e.g.…”

Section: Proposing a Research Strategy For Assessing Understanding An...mentioning

confidence: 99%

Research into evidence-based psychological interventions needs a stronger focus on replicability

Niemeyer,

Knaevelsrud,

van Aert

et al. 2023

Clin. Psychol. Eur.

View full text Add to dashboard Cite

Background It is a precondition for evidence-based practice that research is replicable in a wide variety of clinical settings. Current standards for identifying evidence-based psychological interventions and making recommendations for clinical practice in clinical guidelines include criteria that are relevant for replicability, but a better understanding as well refined definitions of replicability are needed enabling empirical research on this topic. Recent advances on this issue were made in the wider field of psychology and in other disciplines, which offers the opportunity to define and potentially increase replicability also in research on psychological interventions. Method This article proposes a research strategy for assessing, understanding, and improving replicability in research on psychological interventions. Results/Conclusion First, we establish a replication taxonomy ranging from direct to conceptual replication adapted to the field of research on clinical interventions, propose study characteristics that increase the trustworthiness of results, and define statistical criteria for successful replication with respect to the quantitative outcomes of the original and replication studies. Second, we propose how to establish such standards for future research, i.e., in order to design future replication studies for psychological interventions as well as to apply them when investigating which factors are causing the (non-)replicability of findings in the current literature.

show abstract

The logical structure of experiments lays the foundation for a theory of reproducibility

Cited by 10 publications

References 49 publications

Questionable Metascience Practices

Questionable Metascience Practices

Better Accuracy for Better Science . . . Through Random Conclusions

Research into evidence-based psychological interventions needs a stronger focus on replicability

Contact Info

Product

Resources

About