Making literature reviews more reliable through application of lessons from systematic reviews

Haddaway, Neal R.; Woodcock, Paul; Macura, Biljana; Collins, A. J.

doi:10.1111/cobi.12541

Cited by 295 publications

(237 citation statements)

References 33 publications

Supporting

Mentioning

229

Contrasting

Unclassified

Order By: Relevance

“…Ways by which authors of reviews can improve the reliability of their syntheses mainly relate to better reporting and have been detailed extensively elsewhere (e.g. Haddaway et al, 2015;Koricheva and Gurevitch, 2014;Philibert et al, 2012;Roberts et al, 2006). These improvements include aspects such as the development of an apriori protocol, reporting of search strategies and inclusion criteria, and detailing the included studies and extracted data.…”

Section: Discussionmentioning

confidence: 99%

“…CEESAT aims to evaluate review reliability by assessing methodological elements essential for objectivity, transparency and comprehensiveness to enable decision-makers to select reliable, unbiased reviews. Since systematic review methodology was introduced in the environmental sector a decade ago its use has become more widespread (Haddaway et al, 2015). In this context, it is timely to take the opportunity to assess the current reliability of environmental evidence reviews.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The reliability of evidence review methodology in environmental science and conservation

O‘Leary

Kvist

Bayliss

et al. 2016

Environmental Science & Policy

Self Cite

View full text Add to dashboard Cite

Given the proliferation of primary research articles, the importance of reliable environmental evidence reviews for informing policy and management decisions is increasing. Although conducting reviews is an efficient method of synthesising the fragmented primary evidence base, reviews that are of poor methodological reliability have the potential to misinform by not accurately reflecting the available evidence base. To assess the current value of evidence reviews for decision-making we appraised a systematic sample of articles published in early 2015 (N=92) using the Collaboration for Environmental Evidence Synthesis Assessment Tool (CEESAT). CEESAT assesses the methodology of policy-relevant evidence reviews according to elements important for objectivity, transparency and comprehensiveness. Overall, reviews performed poorly with a median score of 2.5/39 and a modal score of zero (range 0-30, mean 5.8), and low scores were ubiquitous across subject areas. In general, reviews that applied meta-analytical techniques achieved higher scores than narrative syntheses (median 18.3 and 2.0 respectively), as a result of the latter consistently failing to adequately report methodology or how conclusions were drawn. However, some narrative syntheses achieved high scores, illustrating that the reliability of reviews should be assessed on a case-by-case basis. Given the potential importance of reviews for informing management and policy, as well as research, it is vital that overall methodological reliability is improved. Although the increasing number of systematic reviews and meta-analyses highlight that some progress is being made, our findings suggest little or no improvement in the last decade. To motivate progress, we recommend that an annual assessment of the methodological reliability of evidence reviews be conducted. To better serve the environmental policy and management communities we identify a requirement for independent critical appraisal of review methodology thus enabling decisionmakers to select reviews that are most likely to accurately reflect the evidence base.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

The reliability of evidence review methodology in environmental science and conservation

O‘Leary

Kvist

Bayliss

et al. 2016

Environmental Science & Policy

Self Cite

View full text Add to dashboard Cite

show abstract

“…A systematic review or map should not be undertaken without specialist methods training if review authors wish to produce a reliable synthesis devoid of major limitations or bias [16,17]. While the major systematic review coordinating bodies have been slow to recognise the benefits of training aimed specifically at stakeholder engagement, as described above, training in the technical aspects of systematic methodology is relatively common (see Box 1 for an example of a recent training course).…”

Section: Systematic Review and Map Methods (Point 2 In Table 2)mentioning

confidence: 99%

“…Training in how to critically appraise reviews can enable stakeholders to highlight common problems with non-systematic reviews. Tools for critical appraisal of reviews have been published for such purposes, for example CEESAT [26], which include assessments of limitations and susceptibility to bias, such as a lack of comprehensiveness and the presence of selection bias and vote-counting [16]. At present, stakeholders may not fully appreciate the potentially fatal characteristics of some non-systematic reviews.…”

Section: Training For Stakeholders Education and Outreachmentioning

confidence: 99%

Much at stake: the importance of training and capacity building for stakeholder engagement in evidence synthesis

Eales¹,

Haddaway²,

Webb³

2017

Environ Evid

View full text Add to dashboard Cite

Systematic reviews and maps are complex methods for synthesising evidence that involve specialist and resourceintensive activities. Systematic reviewers face challenges when attempting to clearly and precisely communicate their methods to end-users and other stakeholder groups. We propose that these challenges are likely to be a key causal factor in the generally low uptake of systematic reviews and maps by policy and practitioners in environmental science and management. We argue that training and capacity building are inherently important components of systematic reviews and maps for all stakeholders; the reviewers themselves, the end-users of specific reviews, and the broader research and decision-making community. Training can help to build capacity for undertaking reviews and maps, and can help to explain complex methods to stakeholders. Training is important for those wishing to undertake stakeholder engagement activities as part of a review. It allows researchers and decision-makers to critique systematic reviews and maps based on their methods. Finally, training may be necessary to allow reviewers to prepare visualisations and communication media for presenting the findings of systematic reviews and maps. We conclude that a broad approach, by viewing every opportunity of stakeholder engagement as a potential for training and capacity building is appropriate both within a specific review and across reviews as a community of practice in evidence synthesis. We call for systematic reviewers to improve networks across disciplines in relation to training, sharing experiences and course content, and ensuring a consistent approach to capacity building in the conduct and use of evidence syntheses.

show abstract

“…Nonetheless, while systematic reviews are becoming more widespread in the environmental sector, not all conform to recognised standards (e.g. [16]) and non-systematic evidence reviews still dominate the review landscape [17]. Moreover, in the medical sector, where systematic review terminology was coined and application of the methodology is most widespread [14,15], the exponential rise in systematic reviews and meta-analyses have been dogged by criticisms that many do not follow full systematic review guidelines, are conflicted by pre-conceived opinions or financial motivations of the authors, and/or have been used to advance industry interests instead of good science [18].…”

Section: Open Accessmentioning

confidence: 99%

Evidence maps and evidence gaps: evidence review mapping as a method for collating and appraising evidence reviews to inform research and policy

et al. 2017

View full text Add to dashboard Cite

Evidence reviews are a key mechanism for incorporating extensive, complex and specialised evidence into policy and practice, and in guiding future research. However, evidence reviews vary in scope and methodological rigour, creating several risks for decision-makers: decisions may be informed by less reliable reviews; apparently conflicting interpretations of evidence may obfuscate decisions; and low quality reviews may create the perception that a topic has been adequately addressed, deterring new syntheses (cryptic evidence gaps). We present a new approach, evidence review mapping, designed to produce a visual representation and critical assessment of the review landscape for a particular environmental topic or question. By systematically selecting and describing the scope and rigour of each review, this helps guide non-specialists to the most relevant and methodologically reliable reviews. The map can also direct future research through the identification of evidence gaps (whether cryptic or otherwise) and redundancy (multiple reviews on similar questions). We consider evidence review mapping a complementary approach to systematic reviews and systematic maps of primary literature and an important tool for facilitating evidence-based decisionmaking and research efficiency.

show abstract

Making literature reviews more reliable through application of lessons from systematic reviews

Cited by 295 publications

References 33 publications

The reliability of evidence review methodology in environmental science and conservation

The reliability of evidence review methodology in environmental science and conservation

Much at stake: the importance of training and capacity building for stakeholder engagement in evidence synthesis

Evidence maps and evidence gaps: evidence review mapping as a method for collating and appraising evidence reviews to inform research and policy

Contact Info

Product

Resources

About