Improving Transparency, Falsifiability, and Rigor by Making Hypothesis Tests Machine-Readable

Lakens, Daniël; DeBruine, Lisa M.

doi:10.1177/2515245920970949

Cited by 30 publications

(31 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Machine-readable hypothesis tests. Whenever one wants to test a prediction using inferential statistics, the involved variables, sample, and evaluation criteria should be specified in unambiguous, standardised codes that allow a computer to evaluate the prediction once the data are in (Lakens & DeBruine, 2021). This procedure ensures a well-defined 'empirical reference' (de Groot, 1969) of the hypothesis.…”

Section: How Can We Move Forward?mentioning

confidence: 99%

“…Widespread adoption of machine-readable hypothesis tests would also complement recent efforts of developmental researchers to build dynamic, community-augmented meta-analyses (Tsuji et al, 2014;Tsuji et al, 2017). An accessible tutorial for making hypothesis tests machine-readable, including a software package and published examples, is provided by Lakens and DeBruine (2021). Helpful instructions for how to move from a hypothesis to an empirical test also can be found in de Groot (1969, esp.…”

Section: How Can We Move Forward?mentioning

confidence: 99%

See 1 more Smart Citation

Why most psychological research findings are not even wrong

Scheel¹

2021

Preprint

View full text Add to dashboard Cite

Psychology’s replication crisis is typically conceptualised as the insight that the published literature contains a worrying amount of unreplicable, false-positive findings. At the same time, meta-scientific attempts to assess the crisis in more detail have reported substantial difficulties to identify unambiguous definitions of the scientific claims in published articles and to determine how they are connected to the presented evidence. I argue that most claims in the literature are so critically underspecified that attempts to empirically evaluate them are doomed to failure — they are not even wrong. Meta-scientists should beware of the flawed assumption that the psychological literature is a collection of well-defined claims. To move beyond the crisis, psychologists must reconsider and rebuild the conceptual basis of their hypotheses before trying to test them.

show abstract

Section: How Can We Move Forward?mentioning

confidence: 99%

Section: How Can We Move Forward?mentioning

confidence: 99%

Why most psychological research findings are not even wrong

Scheel¹

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Increased automation is more and more recognized as a means to improve the research process [54], and therefore this workflow fits well together with other innovations that employ automation, like machine-readable hypothesis tests [55] or automated data documentation [56]. Automated research projects promise a wide range of applications, among them PAC [possibly submitted as a registered report 36,37], Direct Replication [57], fully automated living metanalysis [58], Executable Research…”

Section: Discussionmentioning

confidence: 91%

Reproducible Research in R: A Tutorial on how to Do the Same Thing More Than Once

Peikert¹,

Lissa²,

Brandmaier³

2021

Preprint

View full text Add to dashboard Cite

Reproducibility has long been considered integral to the scientific method. Something is called reproducible when an independent person obtains the same results from the same data. Until recently, detailed descriptions of methods and analyses were the primary instrument for ensuring scientific reproducibility. Technological advancements now enable scientists to achieve a more comprehensive standard; one in which any individual can be granted access to a digital research repository, and reproduce the analyses from the raw data to the final report including all relevant statistical analyses with a single command. This method has far-reaching implications for scientific archiving, reproducibility and replication, scientific productivity, and the credibility and reliability of scientific findings. One obstacle preventing the widespread adoption of this method is that the underlying technological advancements are complicated to use. This paper introduces `repro`, an R-package, which guides researchers in the installation and use of the tools required for making a research project reproducible. Finally, we suggest the use of the proposed tools for the preregistration of study plans as reproducible computer code (preregistration as code; PAC). Since computer code represents the planned analyses exactly as they will be executed, it is more precise than natural language descriptions of those analyses, which merely complement the PAC as a more readable summary. PAC circumvents the shortcomings of ambiguous preregistrations that may give researchers undesired degrees of freedom. Hence, reproducibility made convenient with automation has a wide range of applications to accelerate scientific progress.

show abstract

“…Verbal theory specification leaves room for ambiguities; formalizing our theories with the help of equations or computational models can remove these ambiguities (Smaldino, 2017) and force us to think more carefully about slopes and variances, along with many other assumptions and predictions. Going even further, the resulting statistical hypotheses could be reported in a machine-readable format that results in a maximum of clarity and transparency (Lakens & DeBruine, 2020). But even just a simple data simulation, such as the examples above, can help clarify the relationship between substantive hypothesis and patterns in the data.…”

Section: Choosing the Appropriate Modelmentioning

confidence: 99%

Precise Answers to Vague Questions: Issues With Interactions

Rohrer

Arslan

2021

Advances in Methods and Practices in Psychological Science

View full text Add to dashboard Cite

Psychological theories often invoke interactions but remain vague regarding the details. As a consequence, researchers may not know how to properly test them and may potentially run analyses that reliably return the wrong answer to their research question. We discuss three major issues regarding the prediction and interpretation of interactions. First, interactions can be removable in the sense that they appear or disappear depending on scaling decisions, with consequences for a variety of situations (e.g., binary or categorical outcomes, bounded scales with floor and ceiling effects). Second, interactions may be conceptualized as changes in slope or changes in correlations, and because these two phenomena do not necessarily coincide, researchers might draw wrong conclusions. Third, interactions may or may not be causally identified, and this determines which interpretations are valid. Researchers who remain unaware of these distinctions might accidentally analyze their data in a manner that returns the technically correct answer to the wrong question. We illustrate all issues with examples from psychology and issue recommendations for how to best address them in a productive manner.

show abstract

Improving Transparency, Falsifiability, and Rigor by Making Hypothesis Tests Machine-Readable

Cited by 30 publications

References 19 publications

Why most psychological research findings are not even wrong

Why most psychological research findings are not even wrong

Reproducible Research in R: A Tutorial on how to Do the Same Thing More Than Once

Precise Answers to Vague Questions: Issues With Interactions

Contact Info

Product

Resources

About