Measuring what matters: identifying assessments that reflect learning on the core surgical clerkship

Abstract

Purpose There are various assessments used during the core surgical clerkship (CSC), each of which may be influenced by factors external to the CSC or have inherent biases from an equity lens. In particular, the National Board of Medical Examiners’ Clinical Subject Exams (“Shelf”) is used heavily and may not reflect clerkship curriculum or clinical learning. Methods This is a retrospective review of medical student characteristics and assessments during the CSC from July 2017-June 2021. Assessment methods included: subjective Clinical Performance Assessments (CPA), Shelf, Objective Structured Clinical Examinations, and a short-answer in-house examination (IHE) culminating in a Final Grade (FG) of Honors/Pass/Fail. A Shelf score threshold for Honors was added in academic years 2020–2021. Descriptive, univariate, and multivariable logistic and linear regression statistics were utilized. Results We reviewed records of 192 students. Of these, 107 (55.7%) were female, median age was 24 [IQR: 23–26] years, and most were White/Caucasian ( N = 106, 55.2%). Univariate analysis showed the number of Exceeds Expectations obtained on CPA to be influenced by surgical subspecialty taken ( p = 0.013) and academic year ( p < 0.001). Shelf was influenced by students’ race ( p = 0.009), timing of CSC before or after Internal Medicine (67.9 ± 7.3 vs 72.9 ± 7.1, p < 0.001), and Term taken (increasing from 66.0 ± 8.7 to 73.4 ± 7.5, p < 0.001). IHE scores did not have any external associations. After adjustment with multivariable logistic and linear regressions, CPA and IHE did not have external associations, but higher scores were obtained on Shelf exam in Terms 3, 5, and 6 (by 4.62 [95% CI 0.86–8.37], 4.92 [95% CI 0.53–9.31], and 7.56 [95% CI 2.81–12.31] points, respectively. Odds of FG honors were lower when Shelf threshold was implemented (OR 0.17 [95% CI 0.06–0.50]), and increased as students got older (OR 1.14 [95% CI 1.01–1.30]) or on specific subspecialties, such as vascular surgery (OR 7.06 [95% CI 1.21–41.26]). Conclusions The Shelf is substantially influenced by temporal associations across Terms and timing in relation to other clerkships, such as Internal Medicine. An IHE reflective of a clerkship’s specified curriculum may be a more equitable summative assessment of the learning that occurs from the CSC curriculum, with fewer biases or influences external to the CSC. Supplementary Information The online version contains supplementary material available at 10.1007/s44186-022-00047-8.