It’s a Marathon, Not a Sprint: Rapid Evaluation of Competency-Based Medical Education Program Implementation

Abstract

Purpose Despite the broad endorsement of competency-based medical education (CBME), myriad difficulties have arisen in program implementation. The authors sought to evaluate the fidelity of implementation and identify early outcomes of CBME implementation using Rapid Evaluation to facilitate transformative change. Method Case-study methodology was used to explore the lived experience of implementing CBME in the emergency medicine postgraduate program at Queen’s University, Canada, using iterative cycles of Rapid Evaluation in 2017–2018. After the intended implementation was explicitly described, stakeholder focus groups and interviews were conducted at 3 and 9 months post-implementation to evaluate the fidelity of implementation and early outcomes. Analyses were abductive, using the CBME core components framework and data-driven approaches to understand stakeholders’ experiences. Results In comparing planned with enacted implementation, important themes emerged with resultant opportunities for adaption. For example, lack of a shared mental model resulted in frontline difficulty with assessment and feedback and a concern that the granularity of competency-focused assessment may result in “missing the forest for the trees,” prompting the return of global assessment. Resident engagement in personal learning plans was not uniformly adopted, and learning experiences tailored to residents’ needs were slow to follow. Conclusions Rapid Evaluation provided critical insights into the successes and challenges of operationalizing CBME. Implementing the practical components of CBME was perceived as a sprint, while realizing the principles of CBME and changing culture in postgraduate training was a marathon requiring sustained effort in the form of frequent evaluation and continuous faculty and resident development.

The Transformational Path Ahead: Competency-Based Medical Education in Family Medicine

“…While data gathering on the experience with this new residency design is ongoing, the Queens University team published their initial experience with early implementation, highlighting the iterative, developmental nature and the need for flexibility and adaptation along the journey. 28 Finally, the Macy Foundation published the results of a summit highlighting the potential of time-variable training from both an effectiveness and efficiency perspective, noting training for some learners should be extended beyond traditional norms. 29…”

Evaluation of a National Competency-Based Assessment System in Emergency Medicine: A CanDREAM Study

“…While the investigation of higher-order outcomes in the educational (eg, pursuit of fellowships, etc) and clinical (eg, clinical competence, attending practice metrics, etc) realms has been proposed, 14 substantive variation in the fidelity of the implementation of CBME programs may make it difficult to attribute outcome differences to the assessment program. 7,12 The defining of educationally important and measurable outcomes will be critical for establishing a robust plan for evaluating CBME systems and has been initiated in parallel to this work. 14 Moving forward, we hope to analyze person-level and narrative data.…”

“…[2][3][4] Conversely, local or regional initiatives reveal insights within their context, but it is unclear whether they represent a broader systemic challenge. 2,10,11 Neither type of database is able to detect variability or fidelity of implementation 7,12,13 across individual programs, an essential first step in evaluating higherlevel educational and clinical outcomes. 14 Regardless of the specialty, this is a problem that any program must face when implementing CBME.…”

Seven ways to get a grip on implementing Competency-Based Medical Education at the program level

“…In 2017, 29 specialty programs at Queen's University in Kingston, Canada launched CBME curricula simultaneously, uniquely positioning its program leaders to offer insights to inform others' efforts in their implementation of Competency by Design (CBD) across Canada. 6,7 Through their lived experiences, this cohesive group of postgraduate medical education leaders negotiated a shared understanding of CBME theory and worked collaboratively to navigate the institution-wide transition during the first years of implementation. This collective experience from our university context is shared in a series of tips, adapted from Kotter's 8 step change model, 8 about how to be successful when implementing CBME at the program level.…”