Preliminary Validity Evidence for a Milestones-Based Rating Scale for Chart-Stimulated Recall

Reddy, Shalini; Tekian, Ara; Durning, Steven J.; Gupta, Shanu; Endo, Justin; Affinati, Brenda; Park, Yoon Soo

doi:10.4300/jgme-d-17-00435.1

Cited by 3 publications

(4 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While milestone levels with descriptive anchors are intended to decrease the variability in the evaluation process across individual faculty members and training programs, some level of subjectivity remains. 16 Different CCCs may approach evaluation differently, and specific programs may rate their trainees more or less favorably than other programs. It is unlikely that CCCs considered end-of-residency milestones during their milestone assessments of first-year PEM fellows; if they did, it is unclear how a CCC's knowledge of these scores would affect the assignment of milestone scores in fellowship.…”

Section: Discussionmentioning

confidence: 99%

Milestones comparisons from residency to pediatric emergency medicine fellowship: Resetting expectations

Rose

Shabanova

et al. 2021

AEM Education and Training

View full text Add to dashboard Cite

Background: Pediatric emergency medicine (PEM) fellowships accept trainees who have completed a residency in either emergency medicine (EM) or pediatrics and have adopted 17 subcompetencies with accompanying set of milestones from these two residency programs. This study aims to examine the changes in milestone scores among common subcompetencies from the end of EM or pediatrics residency to early PEM fellowship and evaluates time to reattainment of scores for subcompetencies in which a decline was noted. Methods: This is a national, retrospective cohort study of trainees enrolled in PEM fellowship programs from July 2014 to June 2018. PEM fellowship program directors voluntarily submitted deidentified milestone reports within the study time frame, including end-of-residency reports. Descriptive analyses of milestone scores between end of residency and PEM fellowship were performed. Results: Forty-eight U.S. PEM fellowship programs (65%) provided fellowship milestone data on 638 fellows, 218 (34%) of whom also had end-of-residency milestone scores submitted. Of 218 fellows eligible for analysis, 210 (96%) had completed a pediatrics residency and eight (4%) had completed an EM residency. Pediatric-trained fellows had statistically significant decreases in mean milestone scores in all 10 shared subcompetencies. Reattainment of milestone scores across all common subcompetencies for both EM and pediatric-trained PEM fellows occurred by the end of fellowship. Conclusions: This study demonstrated declines in milestone scores from the end of primary residency training in pediatrics to early PEM fellowship in shared subcompetencies, which may suggest that performance expectations are reset at the beginning of PEM fellowship. Changes in subcompetency milestone anchors to provide subspecialty-specific context may be needed to more accurately define skills acquisition in the residency-to-fellowship transition. How to cite this article: Vu TT, Rose JA, Shabanova V, et al. Milestones comparisons from residency to pediatric emergency medicine fellowship: Resetting expectations. AEM

show abstract

Section: Discussionmentioning

confidence: 99%

Milestones comparisons from residency to pediatric emergency medicine fellowship: Resetting expectations

Rose

Shabanova

et al. 2021

AEM Education and Training

View full text Add to dashboard Cite

show abstract

“…13 This also applies to milestones and to entrustable professional activities. 14,15 Second, although one would hope that criterionreferenced, narrative-based rating scales would facilitate what Reddy et al 7 refer to as shared mental models among raters, as seen in the optimistic findings reported by Crossley et al, 4 improvement in interrater reliability was modest in the current study. In that regard, the results of Reddy and DOI: http://dx.doi.org/10.4300/JGME-D-18-00311.1 colleagues 7 are similar to the experience in employment settings.…”

mentioning

confidence: 55%

“…In this issue of the Journal of Graduate Medical Education, Reddy and colleagues 7 followed that advice in collecting validity data for use with the Accreditation Council for Graduate Medical Education (ACGME) Milestone Projects as narrative anchors for assessment of chart-stimulated recall video scenarios. They found that the 5-point Milestones-Based Rating Scale outperformed a scale rating performance on a ''standard'' 5-point scale ranging from critically deficient to aspirational.…”

mentioning

confidence: 99%

Proceed With Caution: Implementing Competency-Based Graduate Medical Education

Jones¹,

Lockspeiser²

2018

Journal of Graduate Medical Education

View full text Add to dashboard Cite

“…Efforts to address cognitive bias in assessment have included workshops to ‘train out’ variability; revisions of assessment instruments to include detailed instructions, standardised task assessments and competency‐based frameworks; and initiatives to develop shared mental models among assessors. None of these attempts have meaningfully increased assessment validity 5‐7 . Faculty education may help assessors recognise their subjectivity but fails to improve rating inconsistencies, an effect coined the ‘standardisation paradox’ 7 .…”

Section: Introductionmentioning

confidence: 99%

Bias in assessment: name, reframe, and check in

Fainstad

McClintock

Yarris

2021

The Clinical Teacher

View full text Add to dashboard Cite

Cognitive bias permeates almost every learner assessment in medical education. Assessment bias has the potential to affect a learner's education, future career and sense of self‐worth. Decades of data show that there is little educators can do to overcome bias in learner assessments. Using in‐group favouritism as an example, we offer an evidence‐based, three‐step solution to understand and move forward with cognitive bias in assessment: (1) Name: a simple admission about the presence of inherent bias in assessment, (2) Reframe: a rephrasing of assessment language to shed light on the assessor's subjectivity and (3) Check‐in: a chance to ensure learner understanding and open lines of bidirectional communication. This process is theory‐informed and based on decades of educational, sociological and psychological literature; we offer it as a logical first step towards a much‐needed paradigm shift towards addressing bias in learner assessment.

show abstract

Preliminary Validity Evidence for a Milestones-Based Rating Scale for Chart-Stimulated Recall

Cited by 3 publications

References 25 publications

Milestones comparisons from residency to pediatric emergency medicine fellowship: Resetting expectations

Milestones comparisons from residency to pediatric emergency medicine fellowship: Resetting expectations

Proceed With Caution: Implementing Competency-Based Graduate Medical Education

Bias in assessment: name, reframe, and check in

Contact Info

Product

Resources

About