Assessing the evaluability of complex public health interventions: Five questions for researchers, funders and policymakers

Ogilvie, David; Cummins, Steven; Petticrew, Mark; White, Martin; Jones, Andrew; Wheeler, Kathryn

doi:10.1136/jech.2011.143586.6

Cited by 18 publications

(22 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To this end, Ogilvie et al. propose a set of questions researchers, funders and policymakers should use to assess the “evaluability” of complex public health interventions . These questions explore the knowledge that can be generated at different stages of the life cycle of a program along with the strength of that evidence, against what might be practicable to evaluate in the time available.…”

Section: Introductionsupporting

confidence: 84%

“…Within a set budget timeframe or policy framework, there is a need to be pragmatic about which aspects of the program, impacts or outcomes should be included because it is unlikely that everything can be included. Given the unpredictability of the socio‐political context, there may be a need to focus short‐term evaluation efforts more on the theory behind the program, and less on evaluating the program itself . Thus, a comprehensive but practical evaluation may mean focusing an evaluation on high‐priority research questions rather than a detailed evaluation of all program components.…”

Section: Introductionmentioning

confidence: 98%

See 1 more Smart Citation

Applying pragmatic approaches to complex program evaluation: A case study of implementation of the New South Wales Get Healthy at Work program

Crane

Bauman

Lloyd

et al. 2019

Health Prom J of Aust

View full text Add to dashboard Cite

Issue addressed Complex health promotion programs, which can have multilevels of implementation and multi‐components with nonlinear causal pathways, present many evaluation challenges. Traditional evaluation methods often fail to account for the complexity inherent in assessing these programs. In real‐world settings, evaluations of complex programs are often beset by additional constraints of limited budgets and short timeframes. Determining whether a complex program is successful and how a program worked requires evaluators of complex programs to adopt a level of pragmatism. Methods This paper describes a pragmatic evaluation approach used to evaluate the Get Healthy at Work workplace health promotion program, implemented in New South Wales, Australia. Using the program as a case study, we describe some key principles for applying a pragmatic evaluation approach and use these principles to develop an appropriate evaluation strategy. Results The evaluation includes multiple research methods to assess program outputs and implementation; and identify emergent program impacts, within constrained resources. The evaluation was guided by epistemological flexibility, methodological comprehensiveness and operational practicality. Conclusion Health promotion programs, such as state‐wide obesity prevention programs, require appropriate evaluation methods which address their inherent complexity amidst the real‐world evaluation constraints, and focuses on the essential evaluation needs. So what The main complex program evaluation principles are applicable to other multilevel health promotion programs, challenged by methodological and practical or political constraints.

show abstract

Section: Introductionsupporting

confidence: 84%

Section: Introductionmentioning

confidence: 98%

Applying pragmatic approaches to complex program evaluation: A case study of implementation of the New South Wales Get Healthy at Work program

Crane

Bauman

Lloyd

et al. 2019

Health Prom J of Aust

View full text Add to dashboard Cite

show abstract

“…Evaluators are increasingly exhorted to develop diagrammatic logic models and descriptive theories of change 19. These inform the design and conduct of evaluative studies, including the collection of data on the likely causal pathways and the selection of appropriate outcomes 20. However, the development of logic models usually only focuses on the hypothesised intended beneficial impacts of the intervention.…”

Section: Developing ‘Dark Logic’ Models Of Potential Intervention Harmentioning

confidence: 99%

‘Dark logic’: theorising the harmful consequences of public health interventions

Bonell

Jamal

Melendez‐Torres

et al. 2014

J Epidemiol Community Health

268

221

View full text Add to dashboard Cite

Although it might be assumed that most public health programmes involving social or behavioural rather than clinical interventions are unlikely to be iatrogenic, it is well established that they can sometimes cause serious harms. However, the assessment of adverse effects remains a neglected topic in evaluations of public health interventions. In this paper, we first argue for the importance of evaluations of public health interventions not only aiming to examine potential harms but also the mechanisms that might underlie these harms so that they might be avoided in the future. Second, we examine empirically whether protocols for the evaluation of public health interventions do examine harmful outcomes and underlying mechanisms and, if so, how. Third, we suggest a new process by which evaluators might develop ‘dark logic models’ to guide the evaluation of potential harms and underlying mechanisms, which includes: theorisation of agency-structure interactions; building comparative understanding across similar interventions via reciprocal and refutational translation; and consultation with local actors to identify how mechanisms might be derailed, leading to harmful consequences. We refer to the evaluation of a youth work intervention which unexpectedly appeared to increase the rate of teenage pregnancy it was aiming to reduce, and apply our proposed process retrospectively to see how this might have strengthened the evaluation. We conclude that the theorisation of dark logic models is critical to prevent replication of harms. It is not intended to replace but rather to inform empirical evaluation.

show abstract

“…‘Well-being’ is here defined as a eudemonic state in which the individual experiences positive attachment, a sense of meaningfulness and usefulness in life. This framework was used in each area but the delivery method varied according to local needs and priorities, as outlined in current theories concerning the design and evaluation of complex interventions 11 12. Further details are obtainable from the Well London website 13…”

Section: Introductionmentioning

confidence: 99%

‘Well London’ and the benefits of participation: results of a qualitative study nested in a cluster randomised trial

et al. 2014

View full text Add to dashboard Cite

BackgroundWell London is a multicomponent community engagement and coproduction programme designed to improve the health of Londoners living in socioeconomically deprived neighbourhoods. To evaluate outcomes of the Well London interventions, a cluster randomised trial (CRT) was conducted that included a longitudinal qualitative component, which is reported here. The aim is to explore in depth the nature of the benefits to residents and the processes by which these were achieved.MethodsThe 1-year longitudinal qualitative study was nested within the CRT. Purposive sampling was used to select three intervention neighbourhoods in London and 61 individuals within these neighbourhoods. The interventions comprised activities focused on: healthy eating, physical exercise and mental health and well-being. Interviews were conducted at the inception and following completion of the Well London interventions to establish both if and how they had participated. Transcripts of the interviews were coded and analysed using Nvivo.ResultsPositive benefits relating to the formal outcomes of the CRT were reported, but only among those who participated in project activities. The extent of benefits experienced was influenced by factors relating to the physical and social characteristics of each neighbourhood. The highest levels of change occurred in the presence of: (1) social cohesion, not only pre-existing but also as facilitated by Well London activities; (2) personal and collective agency; (3) involvement and support of external organisations. Where the physical and social environment remained unchanged, there was less participation and fewer benefits.ConclusionsThese findings show interaction between participation, well-being and agency, social interactions and cohesion and that this modulated any benefits described. Pathways to change were thus complex and variable, but personal well-being and local social cohesion emerged as important mediators of change.

show abstract

Assessing the evaluability of complex public health interventions: Five questions for researchers, funders and policymakers

Cited by 18 publications

References 0 publications

Applying pragmatic approaches to complex program evaluation: A case study of implementation of the New South Wales Get Healthy at Work program

Applying pragmatic approaches to complex program evaluation: A case study of implementation of the New South Wales Get Healthy at Work program

‘Dark logic’: theorising the harmful consequences of public health interventions

‘Well London’ and the benefits of participation: results of a qualitative study nested in a cluster randomised trial

Contact Info

Product

Resources

About