This article shares methods used to evaluate system cascading failures. A cascading failure occurs when a problem is passed from one subsystem to a downstream subsystem creating a domino effect that undermines system efficiency and effectiveness. First, the basics of system evaluation theory (SET) are reviewed. Then drawing on different examples from the evaluation of emergency response systems the article describes how a) standard operating procedures (SOPs) can be used to locate possible system cascading failure trigger points, and b) mock exercises and secondary data are used to evaluate these trigger points. The discussion highlights the need to expand SET's conceptualization to include within subsystem cascading failures in addition to between subsystem cascading failures. The extent to which program evaluation methods can be adapted for use in system evaluation is also discussed.

No abstract

This article shares lessons learned while evaluating the implementation of a Clinical and Translational Research Centre (CTR). To meet its overarching goals, the CTR consists of numerous research support units (e.g., biostatistics, community engagement, professional development) that are intended to work together collaboratively. It is then argued that an evaluation approach grounded in system thinking was the best fit to evaluate this key CTR design feature. The rationale for selecting systems evaluation theory (SET) as the evaluation framework best suited to evaluate the CTR infrastructure is then presented. The application of SET and the lessons learned are then shared. This article concludes that there are many similarly structured programmes worldwide to which the lessons learned can be applied and upfront investments in using a system approach are rewarded by providing meaningful and useful evaluation recommendations for system change.

The article reports on the inaugural Australian-American system evaluation summit convened in Wyoming, USA, focusing on the application of system evaluation theory (SET). The think tank noted SET’s efficiency principles and published illustrations are emergency response sector specific and pondered whether efficiency measures could be identified for systems targeting complex social problems. The article describes how the think tank used the concept of system waste as a springboard to identify reworks as a universal system efficiency measure. Reworks were defined as repeating all or part of a system standard operating procedure (SOP) or adding additional steps to the SOP to satisfy externalities to the primary purpose. The article then describes the think tank conclusions regarding the utility of reworks as a proxy measure for the four SET factors influencing system efficiency: training, information technology, leadership and culture. The think tank concludes with a discussion of the cautions evaluators should observe when interpreting reworks.

This paper focuses on the application of systems thinking, systems theory, and systems evaluation theory (SET) in evaluating modern day systems. SET consists of three steps purposively sequenced with each being a prerequisite for the success of the next step. The first foundational step is to define the system. Systems thinking provides theoretical rationale for defining the system boundaries, components, and relationships. However, there is no literature describing how to define these system elements. Using an example from the evaluation of several United States cardiac care systems, the paper shares a number of methods used to define the system boundaries, components, and relationships. The paper describes how each of these elements informs the evaluation of step two of SET—evaluating system efficiency. The discussion shares lessons learned, and notes the relationship between methods used in system and program evaluation.