The case for formal methodology in scientific reform

Devezer, Berna; Navarro, Daniel J.; Vandekerckhove, Joachim; Buzbas, Erkan O.

doi:10.1098/rsos.200805

Cited by 123 publications

(97 citation statements)

References 127 publications

(247 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Many voices have been raised calling for theories to be more formal and precise, strengthening the link between theory, construct and hypothesis through mathematical formulation [ 47 – 50 ] and improving publication methods [ 37 ]. Yet, mathematical models are simply abstractions of scientific problems, and thus they can aid scientific inference only to the extent that the abstraction is appropriate to the theory and to the concept-as-intended [ 51 , 52 ].…”

Section: Discussionmentioning

confidence: 99%

When numbers fail: do researchers agree on operationalization of published research?

2021

View full text Add to dashboard Cite

Current discussions on improving the reproducibility of science often revolve around statistical innovations. However, equally important for improving methodological rigour is a valid operationalization of phenomena. Operationalization is the process of translating theoretical constructs into measurable laboratory quantities. Thus, the validity of operationalization is central for the quality of empirical studies. But do differences in the validity of operationalization affect the way scientists evaluate scientific literature? To investigate this, we manipulated the strength of operationalization of three published studies and sent them to researchers via email. In the first task, researchers were presented with a summary of the Method and Result section from one of the studies and were asked to guess the hypothesis that was investigated via a multiple-choice questionnaire. In a second task, researchers were asked to rate the perceived quality of the study. Our results show that (1) researchers are better at inferring the underlying research question from empirical results if the operationalization is more valid, but (2) the different validity is only to some extent reflected in a judgement of the study's quality. These results combined give partial corroboration to the notion that researchers’ evaluations of research results are not affected by operationalization validity.

show abstract

Section: Discussionmentioning

confidence: 99%

When numbers fail: do researchers agree on operationalization of published research?

2021

View full text Add to dashboard Cite

show abstract

“…Multiple scientific fields, cognitive neuroscience included, are faced with the crises of replicability: results of prior experiments often cannot be replicated after being repeated by another or even by the same laboratory. Setting aside the issue of whether replicability is a goal worth pursuing in and of itself (Devezer et al, 2021), the purported crises are a strong indication of substantive issues in the scientific process.…”

Section: Discussionmentioning

confidence: 99%

“…This is in contrast to replicability: obtaining similar experimental results (in terms of hypothesis test results and effect sizes) by using the same experimental design and analysis but applying it to new data. The necessity of replicability is taken for granted by some (Open Science Collaboration, 2015;Pashler & Harris, 2012) and considered more skeptically by others (Devezer, Navarro, Vandekerckhove, & Ozge Buzbas, 2021).…”

Section: Introductionmentioning

confidence: 99%

A Reproducible MEEG Data Analysis Workflow with conda, Snakemake, and R Markdown

Kalenkovich¹,

Levchenko²

2021

Preprint

View full text Add to dashboard Cite

This tutorial is devoted to computational reproducibility, which is an ability to recreate the reported results using the original data and code. Previous studies show that this is impossible for a large percentage of studies with published data and code. We find this situation to be a serious problem for science in general and for the cognitive neuroscience in particular. In this tutorial, we focused on three sources of irreproducibility: differences in software environment, utilization of out-of-date derivative files, and human errors during manual copying of figures, tables, and numbers to the manuscript. We describe three tools that solve these issues: conda, Snakemake, and R Markdown, respectively. Together, they form an eﬀective toolkit that can help researchers achieve reproducibility of their analyses. We demonstrate an application of this toolkit by reimplementing a published data analysis pipeline applied to an open MEEG dataset. Main strengths and weaknesses of our and other approaches are discussed.

show abstract

“…Many voices have been raised calling for theories to be more formal and precise, strengthening the link between theory, construct, and hypothesis through mathematical formulation (47)(48)(49)(50) and improving publication methods (37). Yet, mathematical models are simply abstractions of scientific problems, thus they can aid scientific inference only to the extent that the abstraction is appropriate to the theory and to the concept-as-intended (51,52).…”

Section: Discussionmentioning

confidence: 99%

When numbers fail: Do researchers agree on operationalisation of published research?

Haucke¹,

Hoekstra²,

Ravenzwaaij³

2021

Preprint

View full text Add to dashboard Cite

Current discussions on improving the reproducibility of science often revolve around statistical innovations. However, equally important for improving methodological rigour is a valid operationalisation of phenomena. Operationalisation is the process of translating theoretical constructs into measurable lab-quantities. Thus, the validity of operationalisation is central for the quality of empirical studies. But do differences in the validity of operationalisation affect the way scientists evaluate scientific literature? To investigate this, we adapted three published studies and manipulated the strength of operationalisation. We preregistered our planned study as a registered report. For each study, we designed two tasks in which the validity of the used operationalisation is manipulated and sent them to researchers via email. In the first task, researchers were presented with a summary of the Method and Result section from one of the studies and were asked to guess the hypothesis that was investigated via a multiple-choice questionnaire. In a second task, researchers were asked to rate the perceived quality of the study. Our results show that (1) researchers are better at inferring the underlying research question from empirical results if the operationalisation is more valid; but (2) the different validity is only to some extent reflected in a different judgment of the quality of the study. These results combined give partial evidence for the notion that researchers’ evaluations of research results are not affected by operationalisation validity.

show abstract

The case for formal methodology in scientific reform

Cited by 123 publications

References 127 publications

When numbers fail: do researchers agree on operationalization of published research?

When numbers fail: do researchers agree on operationalization of published research?

A Reproducible MEEG Data Analysis Workflow with conda, Snakemake, and R Markdown

When numbers fail: Do researchers agree on operationalisation of published research?

Contact Info

Product

Resources

About