Developing a Systemic Program Evaluation Methodology: A Critical Systems Perspective

Abstract

In recent years, there has been an increased interest within the program evaluation field in the introduction of systems thinking concepts. However, most of these introductions have been primarily directed towards supporting the practice of evaluation and not towards making theoretical advancements. This article is focused on introducing systems thinking, and specifically perspectives and concepts from the work in critical systems thinking (CST), at a theoretical level in the program evaluation field, towards a reframing of Fourth Generation Evaluation methodology. The process for carrying out such a reframing is introduced, as well as a description of the major changes produced in the evaluation methodology from incorporating the CST perspective. A new model is proposed, and how this model may be beneficial for conducting an evaluation is discussed. Finally, recommendations are made for future developments. © 2018 John Wiley & Sons, Ltd.

Introducing systems‐ and complexity‐informed evaluation

“…Gates (2018) writes about using critical systems heuristics as a way of valuing within evaluations by carefully attending to inclusion, exclusion, and marginalization. Similarly, Torres-Cuello et al (2018) blend fourth-generation evaluation and boundary critique to theorize an evaluation process in which evaluators and stakeholders examine and negotiate the boundaries to be used throughout the evaluative process.…”

Insights and future directions for systems and complexity‐informed evaluation

“…Critical realism has also been used by Mingers and Brocklesby as one of several approaches considering multi‐methodology within critical systems approaches (Jackson, 2019). There have been relatively fewer theoretical developments linking STCS and evaluation practices, yet this has still been a fruitful area of development (Eoyang & Holladay, 2013; Foote et al., 2021; Gates, 2018; Patton, 2011; Torres‐Cuello et al., 2018).…”

Considerations in the Preparation for Teaching Tunnel Engineering

“…As the above mentioned, it is necessary to apply systems thinking in tunnel engineering learning and teaching. Systems thinking is also beneficial in program evaluation design [14]. In general, a well-designed evaluation mode is with the following features: the key points of assessment should match the teaching content; the modes of evaluation are flexible, diverse and incentive; the execution of assessment is planned in the teaching time schedule.…”

“…Systems thinking is applied as a tool, such as in teaching preparation [9], course design [10]. More recently, systems thinking is used in planning course execution procedure in details, such as in terms of test [11], group work [12,13] and program evaluation [14]. But there is no report on the application in the teaching preparation of tunnel engineering.…”