Defining Systems to Evaluate System Efficiency and Effectiveness

Self Cite

This article shares lessons learned while evaluating the implementation of a Clinical and Translational Research Centre (CTR). To meet its overarching goals, the CTR consists of numerous research support units (e.g., biostatistics, community engagement, professional development) that are intended to work together collaboratively. It is then argued that an evaluation approach grounded in system thinking was the best fit to evaluate this key CTR design feature. The rationale for selecting systems evaluation theory (SET) as the evaluation framework best suited to evaluate the CTR infrastructure is then presented. The application of SET and the lessons learned are then shared. This article concludes that there are many similarly structured programmes worldwide to which the lessons learned can be applied and upfront investments in using a system approach are rewarded by providing meaningful and useful evaluation recommendations for system change.

Section: Evaluating Ctr Infrastructure Efficiencymentioning

confidence: 99%

Lessons learned in evaluating the infrastructure of a Centre for Translational Research

Renger¹,

Basson²,

Hart³

et al. 2020

Self Cite

“…Furthermore, significant advances are being made in using system thinking concepts as a foundation for evaluating systems (Renger, 2015, 2016; Renger, Foltysova, Renger, & Booze, 2017; Renger, Keogh, Hawkins, Foltysova & Souvannasacd, 2018). In a series of publications, Renger and his colleagues illustrate how the systems thinking concept of perspectives is used to define the system concepts of boundaries, elements and relationships.…”

Section: System Thinking Concepts Applied To Logic Modelingmentioning

confidence: 99%

The connection between logic models and systems thinking concepts

Renger¹,

Atkinson²,

Renger

et al. 2019

Self Cite

This article examines the relationship between systems thinking concepts and the logic model. Two notable shortcomings of the logic model are illustrated: depicting the program theory linearly and failing to place the program in context. Both issues lead to an artificial depiction of reality making evaluation findings difficult to interpret. Systems thinking concepts are defined and how they address these shortcomings is discussed. This article then demonstrates how many systems thinking concepts are evident in the logic model; although in a limited way. This article concludes by noting the importance of using system concepts when answering evaluation questions related to system dynamics and interrelationships.

“…Currently, SET only notes the importance of detailing the SOP steps when defining a system (Renger et al, 2017). Therefore, one think tank recommendation is to amend SET to include the time standard by which the SOP should be executed.…”

Section: Trainingmentioning

confidence: 99%

“…2 SET consists of three steps: defining the system, evaluating system efficiency and evaluating system effectiveness. In a series of follow-up articles, Renger and his colleagues provided greater detail about how to define systems (Renger, Foltysova, Ienuso, Renger, & Booze, 2017) as well as detailing the evaluation of cascading failures and feedback loops (Renger, 2016; Renger et al, 2017).…”

mentioning

confidence: 99%

Reworks: A robust system efficiency measure

Renger

Keogh²,

Hawkins³

et al. 2018

Self Cite

The article reports on the inaugural Australian-American system evaluation summit convened in Wyoming, USA, focusing on the application of system evaluation theory (SET). The think tank noted SET’s efficiency principles and published illustrations are emergency response sector specific and pondered whether efficiency measures could be identified for systems targeting complex social problems. The article describes how the think tank used the concept of system waste as a springboard to identify reworks as a universal system efficiency measure. Reworks were defined as repeating all or part of a system standard operating procedure (SOP) or adding additional steps to the SOP to satisfy externalities to the primary purpose. The article then describes the think tank conclusions regarding the utility of reworks as a proxy measure for the four SET factors influencing system efficiency: training, information technology, leadership and culture. The think tank concludes with a discussion of the cautions evaluators should observe when interpreting reworks.