Clarifying differences between review designs and methods

Abstract

This paper argues that the current proliferation of types of systematic reviews creates challenges for the terminology for describing such reviews. Terminology is necessary for planning, describing, appraising, and using reviews, building infrastructure to enable the conduct and use of reviews, and for further developing review methodology. There is insufficient consensus on terminology for a typology of reviews to be produced and any such attempt is likely to be limited by the overlapping nature of the dimensions along which reviews vary. It is therefore proposed that the most useful strategy for the field is to develop terminology for the main dimensions of variation. Three such main dimensions are proposed: (1) aims and approaches (including what the review is aiming to achieve, the theoretical and ideological assumptions, and the use of theory and logics of aggregation and configuration in synthesis); (2) structure and components (including the number and type of mapping and synthesis components and how they relate); and (3) breadth and depth and the extent of ‘work done’ in addressing a research issue (including the breadth of review questions, the detail with which they are addressed, and the amount the review progresses a research agenda). This then provides an overarching strategy to encompass more detailed descriptions of methodology and may lead in time to a more overarching system of terminology for systematic reviews.

Systematic review approaches for climate change adaptation research

“…Fewer than 1 % of documents indexed in Web of Knowledge (218 of[60,000) with the title term 'systematic review,' for example, relate to research areas outside of the health or health-related sciences (Supplemental Materials, Search 1). Despite the predominance of systematic reviews in the health sciences, research syntheses guided by systematic methods are prevalent and diverse (Mays et al 2005;Barnett-Page and Thomas 2009;Gough et al 2012). Terminology used to describe literature reviews is murky, however, with different disciplines and publications referring to, for example, scoping or mapping reviews, narrative analysis, and conceptual synthesis, sometimes using systematic methods but differentiated from formal systematic review (Table 1) (Gough et al 2012).…”

“…Despite the predominance of systematic reviews in the health sciences, research syntheses guided by systematic methods are prevalent and diverse (Mays et al 2005;Barnett-Page and Thomas 2009;Gough et al 2012). Terminology used to describe literature reviews is murky, however, with different disciplines and publications referring to, for example, scoping or mapping reviews, narrative analysis, and conceptual synthesis, sometimes using systematic methods but differentiated from formal systematic review (Table 1) (Gough et al 2012). Many reviews employ semi-systematic techniques, often implicitly, but without using the term 'systematic' (Berkhout 2012;McLeman 2013).…”

“…Broadly, a systematic review refers to a focused review of the literature that seeks to answer a specific research question using pre-defined eligibility criteria for documents and explicitly outlined and reproducible methods (Cooper and Hedge 1994;Gough et al 2012). They are distinct from other approaches to research synthesis in that they incorporate an explicit layer of methodological systematization, adding transparency and reproducibility to the review process.…”

“…Standardized criteria for systematic review approaches have been criticized for a presumed bias toward analysis of primarily quantitative data and for their positivist approach to knowledge synthesis, restricting results based on predefined keywords and inclusion/exclusion criteria, and lacking the flexibility of more inductive inquiry approaches (Gough et al 2012;Ansari and Moher 2013). Despite this and other critiques, researchers investigating adaptation have adapted systematic approaches to better meet the needs of their review questions.…”

“…In contrast, increasing interest in systematic approaches to research synthesis of qualitative literature has led to 'middle-ground' approaches and the use of what is often termed meta-synthesis (Jensen and Allen 1996;Walsh and Downe 2005;Merten et al 2010). Meta-synthesis includes a number of more specifically defined approaches, including secondary analysis, grounded meta-analysis, and meta-ethnography (Atkins et al 2008;Suri and Clarke 2009;Barth and Thomas 2012;Gough et al 2012). Many reviews employing a meta-synthesis approach include explicit description of methods for document selection, though apply a more iterative process less strictly defined than the formal inclusion/exclusion criteria endorsed for many quantitative systematic reviews.…”