Tools for Evaluating the Implementation of Complex Education Interventions

Abstract

Large-scale education interventions aimed at diminishing disparities and generating equitable learning outcomes are often complex, involving multiple components and intended impacts. Evaluating implementation of complex interventions is challenging because of the interactive and emergent nature of intervention components. Methods that build from systems science have proven useful for addressing evaluation challenges in the complex intervention space. Complexity science shares some terminology with systems science, but the primary aims and methods of complexity science are different from those of systems science. In this paper we describe some of the language and ideas used in complexity science. We offer a set of priorities for evaluation of complex interventions based on language and ideas used in complexity science and methodologies aligned with the priorities.

Factors associated with implementation of a school-based comprehensive program for students with autism

“…It is important to note that in this study implementation was not perfect nor would one expect it to be. Comprehensive, school-based programs, such as the NPDC program, are similar to complex service programs seen in other disciplines (Kainz et al, 2021; McGaghie, 2011). Implementation consists of multiple features rather than a single implementation outcome such as a fidelity assessment of a single focused intervention practices.…”

“…Given that an index approach is somewhat unique in autism intervention research, a short rationale is provided here. Comprehensive programs that are applied school-wide are like complex social interventions from other disciplines (Kainz et al, 2021) in that they consist of multiple dimensions. The implementation index (IIdx) approach was designed to assess those multiple dimensions.…”

“…Examinations of implementation of comprehensive programs in public schools are unique in at least two ways. First, as Kainz and colleagues (2021) noted, the school context is itself dynamic (e.g. staff turnover and varying schedules) and complex (e.g.…”

Building trusting relationships to support implementation: A proposed theoretical model

“…A theory of change outlines the predicted causal linkages between the activities conducted and the expected short-term, intermediate, and long-term outcomes for the population of interest. Ideally, a theory of change will detail implementation researchers' assumptions about how and why they expect a desired change in implementation outcomes to occur in a particular context (52). The proposed theoretical model is based on findings from our previous research on trust and evidence use, which demonstrates that building trusting relationships and addressing power differentials among stakeholders may be more important than the selection of specific implementation strategies for achieving implementation outcomes (17).…”

Implementation of a multicomponent family support intervention in adult intensive care units: study protocol for an embedded mixed-methods multiple case study (FICUS implementation study)

“… 35 FSIs pose greater challenges for implementation evaluation and cannot rely on pre-established cause and effect specifications. 36 As recommended within implementation science and complex intervention research, the focus should not only lie on the clinical effectiveness of a given intervention, but on the conditions needed to achieve the intervention’s reach and impact in the real-world context. 7 8 Randomised controlled trials are important in establishing intervention effectiveness, however they focus solely on clinical health or service outcomes.…”