Rethinking Reproducibility as a Criterion for Research Quality

Abstract

A heated debate surrounds the significance of reproducibility as an indicator for research quality and reliability, with many commentators linking a "crisis of reproducibility" to the rise of fraudulent, careless and unreliable practices of knowledge production. Through the analysis of discourse and practices across research fields, I point out that reproducibility is not only interpreted in different ways, but also serves a variety of epistemic functions depending on the research at hand. Given such variation, I argue that the uncritical pursuit of reproducibility as an overarching epistemic value is misleading and potentially damaging to scientific advancement. Requirements for reproducibility, however they are interpreted, are one of many available means to secure reliable research outcomes. Furthermore, there are cases where the focus on enhancing reproducibility turns out not to foster high-quality research. Scientific communities and Open Science advocates should learn from inferential reasoning from irreproducible data, and promote incentives for all researchers to explicitly and publicly discuss (1) their methodological commitments, (2) the ways in which they learn from mistakes and problems in everyday practice, and (3) the strategies they use to choose which research component of any project needs to be preserved in the long term, and how.

Rinse and repeat: understanding the value of replication across different ways of knowing

“…Rather than asking what can be achieved in practice, as Collins did, Leonelli [29] Leonelli initiated a taxonomy of replicability, and we expand upon it here. We drew apart her category of non-standard experiments or research based on rare, unique, perishable or inaccessible materials into two: "non-standard experiments" and "research based on rare, unique, perishable or inaccessible materials", since they present different types of limitation on replication and replicability.…”

“…While shared by many, this particular incarnation of this moral imperative is not without its opponents (see e.g. [29]).…”

Finding Wealth in Waste: Irreplicability Re‐Examined

“…When the rigor of conduct and analyses is distrusted, the solution is found in requesting more transparency in research design, data collection, and data analysis so that all research steps become traceable and repeatable, allowing replication. The active pursuit and promotion of open science associated with increased transparency, promoted to filter out sloppy science has uncritically been adopted as the cure for irreplicable research . However, open science's goal cannot and should not be the eradication of irreplicability but rather that of sloppy research.…”

“…The active pursuit and promotion of open science associated with increased transparency, promoted to filter out sloppy science has uncritically been adopted as the cure for irreplicable research. [4] However, open science's goal cannot and should not be the eradication of irreplicability but rather that of sloppy research.…”

Preregistration of analyses of preexisting data

“…Other types of reproducibility are 'inferential reproducibility' and 'indirect reproducibility' (which refer to reproducibility across analytical and experimental procedures). For a more extensive and fine grained discussion of the distinction between these different concepts see Goodman et al (2016) and Leonelli (2018). PREREGISTRATION OF ANALYSES OF PREEXISTING DATA 4 Henrich, 2019), and problematic incentives (Lilienfeld, 2017).…”