Are We Building Our Science on Quicksand? Current Reproducibility Practices in Management

Constructive replication, reproducibility, and generalizability: Getting theory testing for JOMSR right

“…Such a study can, of course, shed light on things like transcription errors, but its primary benefit comes from shedding light on hidden choices made by the original authors. For example, Herndon, Ash, and Pollin (2014) set out to conduct a literal reproducibility study of the Reinhart and Rogoff (2010) examination of the effects of national debt ratios on economic growth (see Cortina et al, 2022, for a more detailed description of both papers). It was only after they were unable to reproduce the original results that they discovered errors, both intentional and unintentional, in the handling and analysis of data by the original authors.…”

“…As such, our first priority for this editorial is to provide clear definitions of the three types of study. In so doing, we draw heavily on our prior work in this space (e.g., Köhler & Cortina, 2021; Cortina, Köhler, & Craig, 2022), which also constitutes the foundations on which the JOMSR Editors based their respective definitions (Kraimer, 2023; Kraimer et al, 2023).…”

“…For example, software packages might use bootstrapping algorithms with different starting points, have different default settings for treating missing data, or treat correlations between residuals differently. This can lead to different results (e.g., Albright & Park, 2009; Asparouhov & Muthen, 2006; Cortina et al, 2022). A reproducibility study can expose any of these sources of variation, thus pointing to the most defensible conclusions vis-à-vis the hypotheses.…”

“…They further argued that the first stage of the mediation would be weaker in Turkey than in the USA because Turkish women would be likely to see harassment at work as normal and not as harassment at all. Cortina et al (2022) reframe this interaction in restricted variance terms: The harassment variable is compressed downwards in Turkey, and this compression reduces the prediction of harassment by job masculinity. But what about the second stage of the mediation?…”

Designing, executing, and publishing replication research: Best practices for successfully taking replication ideas from conceptualization to publication

“…Reproducibility refers to reanalyzing a dataset, whereas replication refers to the reexamination of relationships through the use of a different dataset (Köhler & Cortina, 2021). Literal reproducibility (Cortina et al, 2023) and literal replication (Köhler & Cortina, 2021) refer to studies where researchers attempt to follow the research procedures of the target study as closely as possible. Constructive reproducibility (Cortina et al, 2023) and constructive replication (Köhler & Cortina, 2021) refer to studies in which researchers follow some procedures from the target study, but make strategic modifications to more robustly test a phenomenon or strengthen insights derived from the research.…”