A complex adaptive system approach to evaluation: application to a pay-for-performance program in the USA

Reliability, validity and fairness—key issues in assessing the quality of teaching, instructional leadership and school practice

“…Regarding the first theme on the reliability, validity and fairness of models and methods aiming to evaluate teaching quality while considering design, van der Lans' findings from the Netherlands confirm those of other studies (e.g. Lei et al 2018;Mintrop et al 2018;Santelices et al 2017), in that greater attention has to be paid to the design of such models to provide reliable data and a fair evaluation. The analysis across six value-added models by Sloat et al in the US context demonstrates validity problems, and the authors emphasise that, although the VAM data may be statistically sophisticated, a range of facts, such as the type of tests used, policy contexts and geographical locations, will affect how VAMs play out in practice.…”

Picking Apart a Program Evaluation Committee: A Multiple Case Study Characterizing Primary Care Residency Program Evaluation Committee Structure, Program Improvement, and Outcomes

“…Our case analysis approach to PEC shows the micro day-to-day process and macro level configurations that are essential for promoting ongoing improvements to each program. Mintrop et al (2018) also support that a complex systems lens allows evaluations to “pick up on nuances that other evaluation approaches may overlook” [ 24 ]. Accordingly, our systems approach to PE generates and captures the dynamic and varied program-specific components and processes that must be taken into keen consideration for programmatic improvement [ 24 ].…”

“…Mintrop et al (2018) also support that a complex systems lens allows evaluations to “pick up on nuances that other evaluation approaches may overlook” [ 24 ]. Accordingly, our systems approach to PE generates and captures the dynamic and varied program-specific components and processes that must be taken into keen consideration for programmatic improvement [ 24 ]. Holistic stances counter prevailing reductionist approaches to change that involve dividing complex problems into component parts and are a foundational construct missing in the ACGME competency language [ 22 , 25 , 26 ].…”

Addressing alcohol-related harms in the local night-time economy: a qualitative process evaluation from a complex systems perspective

“…47 A complex systems perspective applied to public health evaluation involves analysing the multiple ways in which a complex system and an intervention interact and influence each other to generate health impacts and health inequalities. [48][49][50][51] Evaluators might consider interventions as 'events' within systems that have the ability to disrupt system behaviour, generating evolving and adaptive patterns of behaviour and emergent outcomes. 44 52 In public health, process evaluations have traditionally been used to understand the mechanisms by which interventions lead to impact, the influence of the broader context on observed variations in impact as well as to assess intervention fidelity and the quality of implementation.…”

“…A complex systems perspective applied to public health evaluation involves analysing the multiple ways in which a complex system and an intervention interact and influence each other to generate health impacts and health inequalities 48–51. Evaluators might consider interventions as ‘events’ within systems that have the ability to disrupt system behaviour, generating evolving and adaptive patterns of behaviour and emergent outcomes 44 52…”