(Re)considering the Concept of Literature Review Reproducibility

Abstract

Literature reviews play a key role in academic research by describing, understanding, explaining, and testing the constructs and theories within a particular topic area. In recent years, various commentaries, debates, and editorials in the information systems (IS) field's top journals have highlighted the importance of a trustworthy literature review process, including detailed discussions on systematicity and transparency. Although the reproducibility of a literature review has also been noted as important, it remains less recognized because of several terminology-related issues. This ambiguity could result in misunderstandings regarding the degree of trust that should be placed in a literature review's process. In this research essay, we seek to clarify what makes a literature review reproducible, how it is distinct from related concepts, and when achieving it is desirable and feasible. We propose a series of clarifications and remedies to assist scholars within and outside the IS field in the preparation of stand-alone reviews.

Repeatability of the searching process in reviews of restoration outcomes

“…In this study, we seek to clarify the concept and applicability of the principle of repeatability by echoing contemporary perspectives from the IS field (Templier & Paré 2018; Cram et al 2020). Further, we used a set of criteria based on Grames and Elphick (2020) and Woodcock et al (2014) for assessing the methodological repeatability of restoration reviews at the searching stage, evaluating whether an independent researcher could repeat review methods, and therefore, to assess the level of trustworthiness researchers and decision‐makers (end‐users) can place in these syntheses, given this is a fundamental review stage that lays the foundation for all successive steps (Johnson & Hennessy 2019).…”

“…An outlook on repeatability, which refers to the methods that can be reperformed by an independent party to generate trust in the methodological process used by the authors has prominently figured in recent publications for literature review in the information science (IS) field (e.g. Boell & Cecez‐Kecmanovic 2015a; Paré et al 2016; Cram et al 2020). In fact, this issue has been claimed to remain less recognized in IS field because of two main factors, which we believe that also apply to the environmental sector (e.g.…”

“…The first factor assumes that there are competing perspectives on the precise definition of repeatability, including inconsistencies relative to other disciplines (e.g. Cram et al 2020). Secondly, the distinctiveness of the repeatability concept may be uncertain and even confounded with others, such as reproducibility and replicability.…”

“…While the former refers to the ability of an independent party to reperform the methods (e.g. the searching strategy) or duplicate the results of an evidence review using an existing dataset (Cram et al 2020), the latter refers to an entirely new study that seeks to corroborate or refute a previous study's results based on independent data collection and analysis (Peng 2011; Dennis & Valacich 2014). The differing perspectives presented by authors across different disciplines may introduce uncertainty into the literature review writing process.…”

“…Definition and applicability of the concepts repeatability, reproducibility, and replicability in each stage of systematic reviews (adapted from Cram et al 2020). Green symbols distinguish between review stages that could benefit from being repeatable, reproducible, and/or replicable.…”