Entrusting Observable Practice Activities and Milestones Over the 36 Months of an Internal Medicine Residency

Abstract

Entrustment of milestones appears to rise progressively over time, with differences by assessor type, competency, milestone, and resident. Further research is needed to elucidate the validity of these data in promotion, remediation, and reporting decisions.

ACGME Milestones Within Subspecialty Training Programs: One Institution's Experience

“…Specifically, in a study assessing internal medicine residency milestones over a 3-year period, the authors noted an increase from 2.46 to 3.92 over a 36-month period of training across all milestones. 20 A similar trend was reported in national Internal Medicine Milestone ratings, 9 which showed an increase up to 0.77 for each PGY level, highlighting a much greater change per PGY than within our cohort of fellows. While recent data from the ACGME show upward trends of milestone ratings in subspecialty training programs nationwide, 17 this does not account for myriad institutional factors that affect milestone ratings.…”

The Promise of Milestones: Are They Living Up to Our Expectations?

“…Based on their findings of limited variability in milestone scores for residents in the same training year, Peabody and colleagues 12 contend that FM Milestones do not measure the amount of inherent ability possessed by a resident, but instead identify where residents are in their progression through residency, and identify residents with lower milestone scores than peers for possible remediation. This study adds to the growing body of literature that provides concurrent validity evidence that residents with higher levels of training have higher milestone scores, 12,[14][15][16]21,22 and lower milestone scores within a postgraduate year level may identify struggling learners. …”

“…14 Several specialties found that milestone ratings differed by subcompetency, [15][16][17] with pediatrics and internal medicine finding that resident professionalism and interpersonal and communication skills were rated highest. 15,16 Response Process: Are Milestone Scores Reliable?…”

Learning Analytics in Medical Education Assessment: The Past, the Present, and the Future

“…However, the systematic application of learning analytics to interpret these data is sporadic. Some groups have started using learning analytics to gather information and gain insights about learner‐ or system‐level performance (e.g., the McMaster Modular Assessment Program for emergency medicine, online modules that using learning analytics methods to teach x‐ray interpration, and an internal medicine program's analytics dashboards). The vast majority of residency programs, however, are attempting to execute programmatic assessment (i.e., the integrated system of multiple, longitudinal observations from multiple observers, aggregated into summary performance scores for group adjudication of global judgment) without optimized data collection (e.g., valid testing/simulations, timely and accurate workplace‐based assessments), modern analytic techniques, or appropriate data representation.…”

“…However, with the digitization of educational records and transition to frequent, direct observation assessments within medical education, databases meeting these criteria are becoming available to medical educators at program, institution, and national levels. In medical education, we rarely have had enough data to use these techniques, but with the increasing use of daily encounter cards, progress testing, observed clinical encounters, milestones, and entrustable professional activity ratings, we are afforded opportunities to now employ learning analytics techniques to guide overall assessment decisions. Importantly, learning analytics offer techniques for examining not only the outcomes of learning but the process, providing information that be used to support trainees and improve their learning.…”