On being ‘Systematic’ in Literature Reviews in IS

Abstract

General guidelines for conducting literature reviews often do not address the question of literature searches and dealing with a potentially large number of identified sources. These issues are specifically addressed by so-called systematic literature reviews (SLRs) that propose a strict protocol for the search and appraisal of literature. Moreover, SLRs are claimed to be a 'standardized method' for literature reviews that is replicable, transparent, objective, unbiased and rigorous, and thus superior to other approaches for conducting literature reviews. These are significant and consequential claims that -despite increasing adoption of SLRs -remained largely unnoticed in the information systems (IS) literature. The objective of this debate is to draw attention of the IS community to SLR's claims, to question their justification and reveal potential risks of their adoption. This is achieved by first examining the origins of SLR and the prescribed SLR process and then by critically assessing their claims and implications. In this debate, we show that SLRs are applicable and useful for a very specific kind of literature review, a meta study that identifies and summarizes evidence from earlier research. We also demonstrate that the claims that SLRs provide superior quality are not justified. More importantly, we argue that SLR as a general approach to conducting literature reviews is highly questionable, concealing significant perils. The paper cautions that SLR could undermine critical engagement with literature and what it means to be scholarly in academic work.

A new paradigm for clinical communication: critical review of literature in cancer care

“…By aggregating literature to answer specific questions, in particular 'what works', systematic reviews tend to perpetuate the paradigm that gives rise to those questions. 38 They are not well suited to questioning the assumptions that led to those questions. Instead we adopted an approach, described by some methodologists as an 'interpretive' or 'critical' review, 39,40 which aims to produce new ideas rather than answer specific questions.…”

“…A new paradigm will not arise from systematic reviews of research literature. By aggregating literature to answer specific questions, in particular ‘what works’, systematic reviews tend to perpetuate the paradigm that gives rise to those questions . They are not well suited to questioning the assumptions that led to those questions.…”

Identifying Quality Factors for Self-Tracking Solutions: A Systematic Literature Review

“…Literature reviews in general "help to identify research problems and gaps and justify the relevance and timeliness of addressing them" [45], while an SLR in particular helps to identify, evaluate, and interpret "all available research relevant to a particular research question, or topic area, or phenomenon of interest" [25]. However, SLRs suffer from certain limitations that make them applicable only under certain conditions: when answering a narrowly defined, summative research question (no "how" or "why" questions) or conducting a bibliometric analysis [4]. Thus, in our case, the SLR is a suitable approach, because our research question is narrow as our focus is on identifying quality factors of self-tracking solutions (not their relationships or possible measurements) and can be answered in a summative form.…”

Searching for evidence or approval? A commentary on database search in systematic reviews and alternative information retrieval methodologies

“…These issues have been found, for example, in SRs on choice architecture, climate vulnerability, and food systems . A study on the use of SR to examine research on technology acceptance found that DBS may systematically exclude relevant literature, sometimes even foundational work . Attempts to correct for terminological instability by including all relevant synonyms will often return unworkably large swathes of irrelevant records.…”