Potential Impact of Pass/Fail Scores on USMLE Step 1: Predictors of Excellence in Obstetrics and Gynecology Residency Training

Abstract

AIM The study aims to determine resident applicant metrics most predictive of academic and clinical performance as measured by the Council of Resident Education in Obstetrics and Gynecology (CREOG) examination scores and Accreditation Council for Graduate Medical Education (ACGME) clinical performance (Milestones) in the aftermath of United States Medical Licensing Examination Scores (USMLE) Step 1 becoming a pass/fail examination. METHODS In this retrospective study, electronic and paper documents for Wayne State University Obstetrics and Gynecology residents matriculated over a 5-year period ending July 2018 were collected. USMLE scores, clerkship grade, and wording on the letters of recommendation as well as Medical Student Performance Evaluation (MSPE) were extracted from the Electronic Residency Application Service (ERAS) and scored numerically. Semiannual Milestone evaluations and yearly CREOG scores were used as a marker of resident performance. Statistical analysis on residents (n = 75) was performed using R and SPSS and significance was set at P < .05. RESULTS Mean USMLE score correlated with CREOG performance and, of all 3 Steps, Step 1 had the tightest association. MSPE and class percentile also correlated with CREOGs. Clerkship grade and recommendation letters had no correlation with resident performance. Of all metrics provided by ERAS, none taken alone, were as useful as Step 1 scores at predicting performance in residency. Regression modeling demonstrated that the combination of Step 2 scores with MSPE wording restored the predictive ability lost by Step 1. CONCLUSIONS The change of USMLE Step 1 to pass/fail may alter resident selection strategies. Other objective markers are needed in order to evaluate an applicant’s future performance in residency.

The Effect of Pass/Fail Exam Grading on Exam Performance in a Pediatric Clerkship

“…18,19 There is a need for additional research to have a better understanding of the actual effects of a pass/fail grading system particularly as Step 1 is now pass/fail which has potential implications for the well-being of medical students and the residency matching process . 20,21…”

Fixing a Broken Clerkship Assessment Process: Reflections on Objectivity and Equity Following the USMLE Step 1 Change to Pass/Fail

“…Despite the emphasis placed on clerkship performance and AΩA honors in residency selection, there is surprisingly little evidence demonstrating that these metrics provide an accurate assessment of clinical performance and/or success in residency. 35–37 For example, AΩA membership and internal medicine clerkship grades were not associated with performance of Accreditation Council for Graduate Medical Education milestones during the intern year in internal medicine. 38 Similar results were found in emergency medicine, internal medicine, surgery, and pediatrics, with AΩA membership and core clerkship grades having no correlation with residency performance.…”

The US Residency Selection Process After the United States Medical Licensing Examination Step 1 Pass/Fail Change: Overview for Applicants and Educators

“…Over 77% of the program directors indicated their belief that this change would make it more difficult to objectively compare candidates [ 27 ]. In some specialties such as neurosurgery, Step 1 scores have been shown to correlate with neurosurgery board exam scores [ 83 ], and similarly, in obstetrics and gynecology, USMLE performance was correlated with that of resident evaluation exams [ 84 ]. In the otorhinolaryngology board exam [ 30 ], underperforming (score<210) was linked to a higher chance of not passing board exams.…”