Use of Emergency Medicine Milestones as Items on End-of-Shift Evaluations Results in Overestimates of Residents' Proficiency Level

Abstract

System, which introduced subcompetencies and the milestones. 1 The Review Committee for Emergency Medicine and the American Board of Emergency Medicine identified 227 emergency medicine (EM) milestones that describe a resident's progression from novice to expert. Each milestone is assigned to 1 of 23 subcompetency areas, and the milestones are associated with a specific developmental stage in resident competency. Residents are expected to reach proficiency (level 4 of the 5-level trajectory) by the end of training. 2,3The ACGME does not specify how programs should assess residents on each milestone. Several programs have incorporated the EM milestones directly into resident assessment, yet there are little data on their reliability, validity, and general use by programs. A recent study found that, when using an assessment tool based on the milestones, 100% of interns met level 1 milestones; yet, when using a standardized observation tool those same interns were competent in only 48% to 93% of the level 1 milestones. 4 To date, no studies have examined the use of milestonebased evaluation tools in end-of-shift evaluations (ESEs), AbstractBackground The emergency medicine milestones were developed to provide more objective resident assessment than current methods. However, little is known about the best practices for applying the milestones in resident assessment.

Building the Plane As We Fly It

“…5,6 Finally, the milestones are formulated to serve as a framework or guide for longitudinal assessment and a group decision process for determining competence, not a focal point in time assessment. 7 Using specific milestones abstracted from that developmental continuum to inform yes/no decisions in the very brief assessment window of a single emergency medicine shift risks loss of these benefits, and may potentially explain the disappointing performance of the ESE reported by Dehon et al 2 The use of milestones as a simple checklist also risks the deconstruction of the highly complex work of patient care. This realization, and the recognition that the resources required to assess all possible milestones are prohibitive, has led to increased interest in entrustmentbased assessment, or entrustable professional activities (EPAs).…”

“…By capturing faculty perceptions (or comments), such as ''felt bad giving a no,'' the authors illuminate some of the pitfalls of this strategy. 2 By providing the full milestone continuum for review, faculty might have selected a milestone more appropriate to the learner's level of competence without the sense that they were making a high-stakes decision or assigning a grade. Ideally, faculty should offer, and learners should embrace, feedback identifying both accomplishments and gaps in performance to advance competence in any given milestone.…”

Critical Appraisal of Emergency Medicine Educational Research: The Best Publications of 2015

“…A total of 434 papers and abstracts satisfied the search criteria, and 61 papers met the inclusion criteria . The authors scored these 61 manuscripts, and the 10 highest scoring quantitative and two qualitative articles are reviewed below, in alphabetical order by first author's last name.…”

“…There were 10 papers (16%) that employed survey methodology but none met the criteria for a highlighted article . The remaining 36 papers (59%) were observational studies and accounted for 50% of the highlighted articles . It is interesting to note that each study design had representation in meeting the validated criteria to be a highlighted article, with the exception, this year, of survey methodology.…”

Designing and Implementing Emergency Department Pain Management Curriculum: A Delphi Approach

“…Currently, many programs use end‐of‐shift evaluations or self‐evaluations to measure proficiency in these milestones. Unfortunately, these two types of evaluation may not accurately reflect the residents’ proficiency level . Both types of measures tend to overestimate proficiency level.…”