Evaluation of a High Stakes Physician Competency Assessment: Lessons for Assessor Training, Program Accountability, and Continuous Improvement

Abstract

Introduction: There is a dearth of evidence evaluating postlicensure high-stakes physician competency assessment programs. Our purpose was to contribute to this evidence by evaluating a high-stakes assessment for assessor inter-rater reliability and the relationship between performance on individual assessment components and overall performance. We did so to determine if the assessment tools identify specific competency needs of the assessed physicians and contribute to our understanding of physician dyscompetence more broadly. Method: Four assessors independently reviewed 102 video-recorded assessments and scored physicians on seven assessment components and overall performance. Inter-rater reliability was measured using intraclass correlation coefficients using a multiple rater, consistency, two-way random effect model. Analysis of variance with least-significant difference post-hoc analyses examined if the mean component scores differed significantly by quartile ranges of overall performance. Linear regression analysis determined the extent to which each component score was associated with overall performance. Results: Intraclass correlation coefficients ranged between 0.756 and 0.876 for all components scored and was highest for overall performance. Regression indicated that individual component scores were positively associated with overall performance. Levels of variation in component scores were significantly different across quartile ranges with higher variability in poorer performers. Discussion: High-stake assessments can be conducted reliably and identify performance gaps of potentially dyscompetent physicians. Physicians who performed well tended to do so in all aspects evaluated, whereas those who performed poorly demonstrated areas of strength and weakness. Understanding that dyscompetence rarely means a complete or catastrophic lapse competence is vital to understanding how educational needs change through a physician's career.

Recasting Assessment in Continuing Professional Development as a Person-Focused Activity

“…This has included, for example, a focus on supporting systems, such as maintenance of certification or revalidation processes, 28,41 as a means of supporting competency-based CPD, 11,42 or to improve quality by overcoming selfassessment challenges. 43,44 Others have focused on descriptions of the types of assessments clinicians participate in, 14,15,45 and how to improve assessment activities (eg, rater performance), 46 standard setting, 47 or on the differences that modalities offer (eg, multiple methods, simulation). 48 There are also validation studies of tools, 49,50 or methods, 51,52 with the aim of doing assessment better.…”

The 2021 Paul Mazmanian Journal of Continuing Education in the Health Professions Award for Excellence in Research

“…Because of the high standard of original research published in JCEHP during 2021, we felt obliged and excited to also share 2 additional honorable mentions. These include a study by René Wong, Simon Kitto, Arno Kumagai, and Cynthia Whitehead for the study titled "Paradox of Patient-Centered Care and the Implication for Patient Involvement in Continuing Professional Development," 3 4 This award is an effort by the journal and partnering organizations, including the Society for Academic Continuing Medical Education, Alliance for Continuing Education in Health Professions, and the Association for Hospital Medical Education, to recognize excellence in original research contributing to the advancement of CPD. The review process has been outlined previously by Kitto et al 5 (2021).…”

“…The authors seek validity evidence to promote the defensibility of or “evidentiary foundations” to support decisions regarding remediation or licensure. The assessment context under study is mainly of physicians who are referred by regulatory authorities or who self-refer “for a variety of reasons, including suspected concerns regarding dyscompetence or physicians seeking to re-enter practice.” 4 The assessment program is robust, and validity evidence is contributed mainly in the form of interrater reliability. Contributing evidence in this space is needed for the purpose of upholding and ensuring social accountabilities and meets the criteria we outlined above for recognition.…”

“…These include a study by René Wong, Simon Kitto, Arno Kumagai, and Cynthia Whitehead for the study titled “Paradox of Patient-Centered Care and the Implication for Patient Involvement in Continuing Professional Development,” 3 and Elizabeth F. Wenghofer, Robert Steele, Richard G. Christiansen, and Misti Carter for the study titled “Evaluation of a High Stakes Physician Competency Assessment: Lessons for Assessor Training, Program Accountability, and Continuous Improvement. 4…”