Medical Knowledge Assessment by Hematology and Medical Oncology In-Training Examinations Are Better Than Program Director Assessments at Predicting Subspecialty Certification Examination Performance

PurposeOn February 12, 2020, the sponsors of the United States Medical Licensing Examination announced that Step 1 will transition to pass/fail scoring in 2022.Step 1 performance has historically carried substantial weight in the evaluation of residency applicants and as a predictor of subsequent subject-specific medical knowledge. Using a systematic review and meta-analysis, the authors sought to determine the association between Step 1 scores and in-training examination (ITE) performance, which is often used to assess knowledge acquisition during residency. MethodThe authors systematically searched Medline, EMBASE, and Web of Science for observational studies published from 1992 through May 10, 2020. Observational studies reporting associations between Step 1 and ITE scores, regardless of medical or surgical specialty, were eligible for inclusion. Pairs of researchers screened all studies, evaluated quality assessment using a modified Newcastle-Ottawa Scale, and extracted data in a standardized fashion. The primary endpoint was the correlation of Step 1 and ITE scores. ResultsOf 1,432 observational studies identified, 49 were systematically reviewed and 37 were included in the meta-analysis. Overall study quality was low to moderate. The pooled estimate of the correlation coefficient was 0.42 (95% confidence interval [CI]: 0.36, 0.48; P < .001), suggesting a weak-to-moderate positive correlation between Step 1 and ITE scores. The random-effects meta-regression found the association between Step 1 and ITE scores was weaker for surgical (versus medical) specialties (beta -0.25 [95% CI: -0.41, -0.09; P = .003]) and fellowship (versus residency) training programs (beta -0.25 [95% CI: -0.47, -0.03; P = .030]). ConclusionsThe authors identified a weak-tomoderate positive correlation betweenStep 1 and ITE scores based on a meta-analysis of low-to-moderate quality observational data. With Step 1 scoring transitioning to pass/fail, the undergraduate and graduate medical education communities should continue to develop better tools for evaluating medical students.

show abstract

“…Thirty-seven studies were ultimately included in our data extraction and subsequent pooled analysis (see Figure 1). 10–46…”

Section: Resultsmentioning

confidence: 99%

Association Between USMLE Step 1 Scores and In-Training Examination Performance: A Meta-Analysis

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Test reliability is high (KR 20 >0.90) across administrations similar to national certifying exams [6]. In-training examination (ITE) scores have been reported to be better predictors of subsequent resident scores on subspecialty certification exams than ratings of medical knowledge from program directors [7]. Since the IM-ITE is a national, standardized exam residents take every year, the IM-ITE can serve as a measure of both knowledge of general IM and IM subspecialties.…”

Section: Introductionmentioning

confidence: 99%

Does an endocrinology subspecialty residency rotation enhance resident endocrine clinical knowledge?

et al. 2022

View full text Add to dashboard Cite

Background Internal Medicine (IM) programs offer elective subspecialty rotations in which residents may enroll to supplement the experience and knowledge obtained during general inpatient and outpatient rotations. Objective evidence that these rotations provide enhanced subspecialty specific knowledge is lacking. The purpose of this study was to determine whether exposure to an endocrinology subspecialty rotation enhanced a resident’s endocrinology-specific knowledge beyond that otherwise acquired during IM residency. Methods Data were collected on internal medicine resident scores on the American College of Physicians Internal Medicine In-Training Examinations (IM-ITE) for calendar years 2012 through 2018 along with enrollment data as to whether residents had completed an endocrinology subspecialty rotation prior to sitting for a given IM-ITE. Three hundred and six internal medicine residents in the University of Minnesota Internal Medicine residency program with 664 scores total on the IM-ITE for calendar years 2012 through 2018. Percentage of correct answers on the overall and endocrine subspecialty content areas on the IM-ITE for each exam were determined and the association between prior exposure to an endocrinology subspecialty rotation and percentage of correct answers in the endocrinology content area was analyzed using generalized linear mixed-effects models. Results Two hundred and thirty-three residents (76%) completed an endocrinology subspecialty rotation at some point during their residency; 121 (40%) residents had at least one IM-ITE both before and after exposure to an endocrine subspecialty rotation. Exposure to an endocrinology subspecialty rotation exhibited a positive association with the expected IM-ITE percent correct on the endocrinology content area (5.5% predicted absolute increase). Advancing year of residency was associated with a predicted increase in overall IM-ITE score but did not improve the predictive model for endocrine subspecialty score. Conclusions Completion of an endocrinology subspecialty elective was associated with an increase in resident endocrine specific knowledge as assessed by the IM-ITE. These findings support the value of subspecialty rotations in enhancing a resident’s subspecialty specific medical knowledge.

show abstract

“…Studies from specialties such as hematology, infectious disease, and nephrology have shown that there are positive associations between ITE scores and board CEs scores, and ITE performance may be used to predict candidates' likelihood of success in a future certification attempt. [1][2][3] The need for an ITE for medical genetics and genomics residency programs was first discussed at the 2009 meeting of the Residency Program Directors. A small committee was formed to develop and implement the examination, which was administered from 2010 to 2012 through the University of Alabama at Birmingham.…”

Section: Introductionmentioning

confidence: 99%

The relationship between performance on the medical genetics and genomics in-training and certifying examinations

Waggoner

O’Donnell

Ligon

et al. 2022

Genetics in Medicine

View full text Add to dashboard Cite

The American Board of Medical Genetics and Genomics (ABMGG) certifying examinations (CEs) are designed to assess relevant basic knowledge, clinical knowledge, and diagnostic skills of board-eligible candidates in primary specialty areas. The ABMGG intraining examinations (ITEs) provide formative feedback regarding knowledge and learning over time and assess readiness to attempt board certification. This study addresses the validity of the ABMGG ITE by evaluating its relationship with performance on CE utilizing established psychometric approaches. Methods: Statistical analysis included bivariate Pearson correlation coefficients and linear regression to evaluate the strength of associations between ITE scores and CE scores. Logistic regression was used to assess the association between ITE scores and the probability of passing each CE. Results: Logistic regression results indicated that ITE scores accounted for 22% to 44% of the variability in CE outcomes. Across 3 certification cycles, for every 1-point increase in ITE scores, the odds ratio for earning a passing score increased by a factor of 1.12 to 1.20 for the general CE, 1.14 to 1.25 for the clinical CE, and 1.12 to 1.20 for the laboratory CEs. Conclusion:The findings show a positive correlation between performance on the ITE examination and performance on and passing the ABMGG CE.

show abstract

Medical Knowledge Assessment by Hematology and Medical Oncology In-Training Examinations Are Better Than Program Director Assessments at Predicting Subspecialty Certification Examination Performance

Cited by 11 publications

References 15 publications

Association Between USMLE Step 1 Scores and In-Training Examination Performance: A Meta-Analysis

Association Between USMLE Step 1 Scores and In-Training Examination Performance: A Meta-Analysis

Does an endocrinology subspecialty residency rotation enhance resident endocrine clinical knowledge?

The relationship between performance on the medical genetics and genomics in-training and certifying examinations

Contact Info

Product

Resources

About