Assessment methods and resource requirements for milestone reporting by an emergency medicine clinical competency committee

Abstract

Background: The Accreditation Council for Graduate Medical Education (ACGME) introduced milestones for Emergency Medicine (EM) in 2012. Clinical Competency Committees (CCC) are tasked with assessing residents on milestones and reporting them to the ACGME. Appropriate workflows for CCCs are not well defined.Objective: Our objective was to compare different approaches to milestone assessment by a CCC, quantify resource requirements for each and to identify the most efficient workflow.Design: Three distinct processes for rendering milestone assessments were compared: Full milestone assessments (FMA) utilizing all available resident assessment data,Ad-hoc milestone assessments (AMA) created by multiple expert educators using their personal assessment of resident performance,Self-assessments (SMA) completed by residents.FMA were selected as the theoretical gold standard. Intraclass correlation coefficients were used to analyze for agreement between different assessment methods. Kendall’s coefficient was used to assess the inter-rater agreement for the AMA.Results: All 13 second-year residents and 7 educational faculty of an urban EM Residency Program participated in the study in 2013. Substantial or better agreement between FMA and AMA was seen for 8 of the 23 total subcompetencies (PC4, PC8, PC9, PC11, MK, PROF2, ICS2, SBP2), and for 1 subcompetency (SBP1) between FMA and SMA. Multiple AMA for individual residents demonstrated substantial or better interobserver agreement in 3 subcompetencies (PC1, PC2, and PROF2). FMA took longer to complete compared to AMA (80.9 vs. 5.3 min, p < 0.001).Conclusions: Using AMA to evaluate residents on the milestones takes significantly less time than FMA. However, AMA and SMA agree with FMA on only 8 and 1 subcompetencies, respectively. An estimated 23.5 h of faculty time are required each month to fulfill the requirement for semiannual reporting for a residency with 42 trainees.

Resident Self-assessment and Clinical Competency Committee Evaluations in Family Medicine: A Single-Institution Study

“…5 Self-assessment has the potential to address these challenges 6 and is emerging as an important tool in medical education because of its potential to align learner/instructor expectations, 7,8 catalyze self-reflection, 8 identify program deficiencies and guide curriculum development, 9 and encourage lifelong habits of accurate self-assessment driving self-directed learning and improvement. 7,10,11 Several non-primary care oriented residency programs have introduced self-assessment into milestone assessments [12][13][14][15][16][17][18][19][20][21][22][23] ; however, we have not seen milestone selfassessment introduced in a family medicine residency program in the US literature. This article shares the development of a self-assessment process for family medicine residents and the findings of a comparison between CCC assessments and resident self-assessments.…”