Systematic reviews of health effects of social interventions: 2. Best available evidence: how low should you go?

Abstract

Study objective: There is little guidance on how to select the best available evidence of health effects of social interventions. The aim of this paper was to assess the implications of setting particular inclusion criteria for evidence synthesis. Design: Analysis of all relevant studies for one systematic review, followed by sensitivity analysis of the effects of selecting studies based on a two dimensional hierarchy of study design and study population. Setting: Case study of a systematic review of the effectiveness of interventions in promoting a population shift from using cars towards walking and cycling. Main results: The distribution of available evidence was skewed. Population level interventions were less likely than individual level interventions to have been studied using the most rigorous study designs; nearly all of the population level evidence would have been missed if only randomised controlled trials had been included. Examining the studies that were excluded did not change the overall conclusions about effectiveness, but did identify additional categories of intervention such as health walks and parking charges that merit further research, and provided evidence to challenge assumptions about the actual effects of progressive urban transport policies. Conclusions: Unthinking adherence to a hierarchy of study design as a means of selecting studies may reduce the value of evidence synthesis and reinforce an ''inverse evidence law'' whereby the least is known about the effects of interventions most likely to influence whole populations. Producing generalisable estimates of effect sizes is only one possible objective of evidence synthesis. Mapping the available evidence and uncertainty about effects may also be important. D espite increasing calls for systematic reviews of health effects of social interventions, there is little methodological research or even guidance on how such reviews should be done. We have lifted the lid on the ''private life'' of the input side of one such systematic review to expose some of our methodological processes and decisions to critical analysis.1 In a companion paper, we set the scene and examined one phase of the review, the search for evidence. 2In this paper, we examine another phase of the review: the selection of evidence for inclusion. We investigate the effect of varying our inclusion criteria on the findings and overall value of the review. SELECTING EVIDENCE FOR INCLUSIONResearchers designing systematic reviews of intervention studies are advised to specify their research questions in terms of four facets: the intervention, the population receiving the intervention, the outcome of interest, and the study designs deemed worthy of inclusion.3 This approach is undoubtedly helpful for structuring research questions and protocols, but we aimed to synthesise population level evidence in a cross disciplinary field where comparatively little empirical intervention research has been done. A broad understanding of population health and its wider determinants implied a...

Using qualitative metasummary to synthesize qualitative and quantitative descriptive findings

“…Scholars have increasingly argued against the a priori exclusion of studies for reasons of quality (Conn & Rantz, 2003;Cooper, 1998;Higgins & Green, 2005, Section 6) or ''censor(ship) by some a priori set of prejudices'' (Glass, 2000, p. 10), and for the use of typologies (as opposed to fixed hierarchies) of evaluation distinctively appropriate to them (Ogilvie, Egan, Hamilton, & Petticrew, 2005;Petticrew & Roberts, 2003). Moreover, reports and the findings in them are more or less suitable for inclusion depending on the nature of the question asked and type of analysis conducted.…”

Evidence-Based Public Health: A Fundamental Concept for Public Health Practice

“…We define four categories within a typology of scientific evidence for decision making: evidence-based, efficacious, promising, and emerging interventions. Adherence to a strict hierarchy of study designs may reinforce an inverse evidence law by which interventions most likely to influence whole populations (e.g., policy change) are least valued in an evidence matrix emphasizing randomized designs (125,127). Type 3 evidence (of which we have the least) shows how and under which contextual conditions interventions were implemented and how they were received, thus informing "how something should be done" (141).…”

La investigación traslacional y su aporte para la toma de decisiones en políticas de salud

“…y como respuesta a la complejidad obvia asociada a este tipo de síntesis de evidencia, han surgido nuevas concepciones teóricas que intentan abordarlas, como la evaluación realista. Este tipo de revisión y síntesis, generalmente crucial para políticas de salud, se centra en responder qué funciona, en quiénes funciona y bajo qué circunstancias funciona (45)(46)(47)(48) .…”