Aiming for Utility in ‘Systems-based Evaluation’: A Research-based Framework for Practitioners

Abstract

The publication of this IDS Bulletin is evidence that there is a growing philosophy, debate and a quest for examples showing that the application of systems thinking and related approaches could provide improved processes, tools and insights for evaluations and their managers and clients. However, the utility of SBE approaches has not been rigorously studied through case examples for the purpose of providing practical recommendations. SBE is the application of specific methods that are influenced by systems concepts and methodologies to the task of evaluation. This article proposes a set of recommendations for practice for evaluators and managers aiming to employ techniques derived from, or influenced by, a systems perspective.SDM was used and process researched from 2006 to 2008 as a case to investigate its utility as an SBE approach. An SD model was developed to ex ante evaluate the potential requirements and implications on the health systems of the ambitious ART scale-up strategy in Lusaka, Zambia that was being developed and implemented during that period. SD was chosen, with input from the Ministry of Health, as it had been used previously for public health and disease control programme planning and evaluation and provided an accessible literature and a cadre of professional consulting resources (Homer and St Clair 1991;Homer et al. 2004;Homer and Hirsch 2006;Homer et al. 2010;Ritchie-Dunham and Mendez-Galvan 1999;Sterman 2006;Tebbens et al. 2005;Thompson and Tebbens 2010).This article lays out the research stance, process and description of the case. Key features of the SD model and process are presented and described. Exemplary quotations from the research conversations before and after the SD model development among stakeholders are shared. An operational definition of SBE is proposed as well as a framework for utility as emerged from the research. Finally, implications of the research are discussed and conclusions for broader practice of SBE are proposed.This article is based on a dissertation project cited herein and available at http://researchbank.rmit.edu.au/view/rmit:160050. 58Aiming for Utility in 'Systems-based Evaluation': A Research-based Framework for Practitioners John T. GroveAbstract System dynamics modelling (SDM) was used and process researched as a case to investigate its utility as a systems-based evaluation (SBE) approach. A system dynamics (SD) model 1 was developed to evaluate the potential requirements and implications on the health systems of the ambitious antiretroviral therapy (ART) scale-up strategy in Lusaka, Zambia. Research on SDM for strategic evaluation provided insights and principles for future application of SBE. The SD diagrams readily inspired new insights while practical constraints limited use of the model for action planning. Research suggests that utility of SBE begins with engaging stakeholders to share and align their views on a representation of the system and progresses to their reinterpretations of the system that they inhabit, ultimately moving towards trans...

Tools for Supporting the MCH Workforce in Addressing Complex Challenges: A Scoping Review of System Dynamics Modeling in Maternal and Child Health

“…The disease epidemics most commonly addressed were related to STIs. Increasing understanding (Defined as using a model to increase scientific understanding of a given topic) 55 (Heidenberger & Flessa, 1993, Rauner, 2002, Maital & Bornstein, 2003, Staller, 2004, Fredericks et al, 2008, Moxnes & Jensen, 2009, Rwashana et al, 2009, Schrottner, Konig et al 2009, Finegood et al, 2010, Bridgewater et al, 2011, Hamdani et al, 2011, Osgood, Dyck, et al, 2011Osgood, Mahamoud, et al, 2011, Osgood, Dyck, et al, 2011Osgood, Mahamoud, et al, 2011, Siegel et al, 2011, BeLue et al, 2012, Davison, Vanderwater et al 2012, Ghaffarzadegan et al, 2013, Patrick et al, 2013, Weeks et al, 2013, Fallah-Fini et al, 2014, Gillen et al, 2014, Kumar & Kumar, 2014, Lan, Chen et al 2014, Minyard et al, 2014, Rwashana, Nakubulwa et al 2014, Sabounchi et al, 2014, Batchelder et al, 2015, Frerichs et al, 2015, Grove, 2015, Hoehner et al, 2015, Keane et al, 2015, Martin et al, 2015a, 2015b, 2015c, M...…”

Participatory systems mapping for complex energy policy evaluation

“…Barnes et al, 2003; Sanderson, 2000), to examples of methods being applied (e.g. Qualitative Comparative Analysis, Befani, 2013; Byrne, 2013; and causal loop diagrams and system dynamics, Dyehouse et al, 2009; Fredericks et al, 2008; Grove, 2015), to reflections on the success and value of complexity’s application (e.g. Gates, 2016; Mowles, 2014; Reynolds et al, 2016; Walton, 2014; Williams, 2015).…”

“…More closely related to the Participatory Systems Mapping approach we present here are causal loop diagrams and Systems Dynamics (e.g. Dyehouse et al, 2009; Fredericks et al, 2008; Grove, 2015), and Bayesian Belief Networks (sometimes known as ‘dependency modelling’) (Brand et al, 2015; Sedighi and Varga, 2019). Fuzzy Cognitive Mapping (e.g.…”

Policy evaluation for a complex world: Practical methods and reflections from the UK Centre for the Evaluation of Complexity across the Nexus

“…There are many specific complexity-appropriate methods presented in the evaluation literature, including agent-based modelling (Morell et al, 2010), case study approaches (Woolcock, 2013), causal loop diagrams and Systems Dynamics (Dyehouse et al, 2009; Fredericks et al, 2008; Grove, 2015), Process tracing (Schmitt and Beach, 2015), Qualitative Comparative Analysis (Befani, 2013; Blackman, 2013; Blackman et al, 2013; Byrne, 2013), social network analysis (Drew et al, 2011; Durland and Fredericks, 2005), and co-produced or exploratory approaches to working with stakeholders (Copestake, 2014). Moving towards 2020 we get closer to the some of the concerns of this special issue, as scholars begin to become more critical and self-reflexive about the value and use of these ideas and methods, particularly in terms of the realities of using them in the practice of evaluation (e.g.…”