The reliability of evidence review methodology in environmental science and conservation

Abstract

Given the proliferation of primary research articles, the importance of reliable environmental evidence reviews for informing policy and management decisions is increasing. Although conducting reviews is an efficient method of synthesising the fragmented primary evidence base, reviews that are of poor methodological reliability have the potential to misinform by not accurately reflecting the available evidence base. To assess the current value of evidence reviews for decision-making we appraised a systematic sample of articles published in early 2015 (N=92) using the Collaboration for Environmental Evidence Synthesis Assessment Tool (CEESAT). CEESAT assesses the methodology of policy-relevant evidence reviews according to elements important for objectivity, transparency and comprehensiveness. Overall, reviews performed poorly with a median score of 2.5/39 and a modal score of zero (range 0-30, mean 5.8), and low scores were ubiquitous across subject areas. In general, reviews that applied meta-analytical techniques achieved higher scores than narrative syntheses (median 18.3 and 2.0 respectively), as a result of the latter consistently failing to adequately report methodology or how conclusions were drawn. However, some narrative syntheses achieved high scores, illustrating that the reliability of reviews should be assessed on a case-by-case basis. Given the potential importance of reviews for informing management and policy, as well as research, it is vital that overall methodological reliability is improved. Although the increasing number of systematic reviews and meta-analyses highlight that some progress is being made, our findings suggest little or no improvement in the last decade. To motivate progress, we recommend that an annual assessment of the methodological reliability of evidence reviews be conducted. To better serve the environmental policy and management communities we identify a requirement for independent critical appraisal of review methodology thus enabling decisionmakers to select reviews that are most likely to accurately reflect the evidence base.

Evidence maps and evidence gaps: evidence review mapping as a method for collating and appraising evidence reviews to inform research and policy

“…For this purpose we recommend the Collaboration for Environmental Evidence Synthesis Assessment Tool (CEESAT; [19]). The current version of CEESAT (available at http://www.environmentalevidence.org/review-appraisals) consists of 13 criteria relating to the reliability (combining objectivity, transparency, and comprehensiveness) of reviews (see [19] for details), and achieves good repeatability when independent assessments of the same review are compared [6,12,19]. For each criterion, reviews receive 3 points, 1 point, or 0 points.…”

“…However, as with primary research, reviews can differ in the rigour of the methods and the reliability of findings (e.g. [6,[9][10][11][12]), and subtle differences in scope may influence their applicability to a particular problem. Indeed, the majority of reviews in environmental science are not conducted according to predefined guidelines, but instead apply a range of methods that each promote or compromise reliability to varying degrees (e.g.…”

“…Indeed, the majority of reviews in environmental science are not conducted according to predefined guidelines, but instead apply a range of methods that each promote or compromise reliability to varying degrees (e.g. [6]). Differences in methodological reliability amongst reviews on similar topics create several related risks for researchers and decision-makers:…”

“…narrative/qualitative (which may include limited quantitative analyses) or metaanalyses; see [6] be objectively categorised is influenced by the methods employed in the review. Whilst a meta-analysis can usually be objectively categorised as addressing a particular question based on whether or not effect sizes are presented, there is no such obvious distinction in many narrative reviews, in which questions could be addressed through varying amounts of text with varying degrees of relevance and supporting references.…”

“…decision-makers and researchers in other fields) are increasingly likely to rely on evidence reviews that synthesise evidence across the spectrum of primary literature related to a specific, policy-relevant question [3][4][5][6]. Evidence reviews (hereafter also referred to as 'reviews') attempt to answer a specific question by aggregating and synthesising the results of primary studies and may include meta-analysis (statistical methods for combining the magnitude of the outcomes [effect sizes] across different data sets addressing the same research question [7]) and/or narrative synthesis (use of prose to summarise and draw conclusions from primary research which may be supplemented by the reviewers' own experience and may include limited quantitative analysis [6]). Evidence…”