Clinical Scholars: Using Program Evaluation to Inform Leadership Development

Abstract

Leadership development programs are notoriously difficult to evaluate, and when evaluations are attempted, they often do not go beyond measuring low-level, short-term outcomes of the impacts experienced by participants. Many leadership development programs do not systematically assess changes that are catalyzed within the organizations, communities and systems in which participants lead. To address these challenges, evaluators of the Clinical Scholars National Leadership Institute (CNLI) have designed a comprehensive, mixed-methods evaluation approach to determine the effectiveness of the training and explore the impacts of participants in the spheres in which they lead. Guided by Michael Patton’s Developmental Evaluation approach and framed by Kirkpatrick’s Training Evaluation Model, the CSNLI evaluation collects data on multiple levels to provide a robust picture of the multiple outcomes of the program. The approach focuses on individual participant outcomes, by measuring competency changes over time and exploring how participants use the competencies gained through the training in their work. Social network analysis is utilized to measure the development and expansion of participants’ networks and collaboration within the teams, cohorts, and across sectors and disciplines throughout their time in the CSNLI. The Most Significant Change methodology and semi-structured alumni interviews are used to measure impacts participants identify as occurring as a result of their participation. Finally, Concept Mapping is implemented to explore how Fellows make meaning of the foundational concepts and values of the CSNLI. The outcome and impact evaluation activities employed by the CSNLI, in combination with quality improvement-focused process evaluation, support innovation and excellence in the provision of a health equity-grounded leadership development program.

Advancing Health Equity Through Equity-Centered Leadership Development with Interprofessional Healthcare Teams

“…A detailed description of the CS evaluation approach is presented elsewhere. 16 For the purposes of this paper, we explore Kirkpatrick's model levels 2-4, as defined below: Level 2: Learning—participants’ gains in knowledge, self-efficacy, skills, and attitudes of EDI-focused competencies Level 3: Behavior—translation of gained knowledge and skills into tangible EDI-relevant actions Level 4: Results—EDI-relevant impacts on participants’ individual leadership, organizations, and communities …”

Evaluation of a leadership development impact assessment toolkit: a comparative case study of experts’ perspectives in three Canadian provinces

“…As a result, the evaluation of the programs was frequently reported as complex and challenging often failing to connect to key outcomes (Phillips et al , 2015a). Specifically, the literature highlights the difficulty of measuring the impact of complex interventions (Streatfield and Markless, 2019; Davys et al , 2017; Dave et al , 2021; Moldoveanu and Narayandas, 2019; Hopkins, and Meyer, 2019) and on moving beyond process-type of evaluation (Dave et al , 2021; Davys et al , 2017). Challenges include but are not limited to the lack of evaluation skills and resources for organizations (Davys et al , 2017), a paucity of rigorous evaluations of LD programs to use as comparisons (Joseph-Richard and McCray, 2022) and difficulty in quantifying the return on investment (ROI) for programs (Day et al , 2021).…”

“…Challenges include but are not limited to the lack of evaluation skills and resources for organizations (Davys et al , 2017), a paucity of rigorous evaluations of LD programs to use as comparisons (Joseph-Richard and McCray, 2022) and difficulty in quantifying the return on investment (ROI) for programs (Day et al , 2021). Also critical is the need for more comprehensive multimethod, multilevel, systematic and robust evaluations including a focus on outcome and impact (Collins and Denyer, 2008; Dave et al , 2021; Jeyaraman et al , 2018; Newstead et al , 2020; Njah et al , 2021; Wallace et al , 2021). This ongoing challenge was also stressed in a King's Fund review, which highlighted the lack of literature on the effectiveness of the LD programs and called for the need for evidence-based approaches to ensure ROI (West et al , 2015).…”