Validity Evidence for Medical School OSCEs: Associations With USMLE<sup>®</sup>Step Assessments

Dong, Ting; Swygert, Kimberly A.; Durning, Steven J.; Saguil, Aaron; Gilliland, William R.; Cruess, David F.; DeZee, Kent J.; LaRochelle, Jeffrey; Artino, Anthony R.

doi:10.1080/10401334.2014.960294

Cited by 33 publications

(23 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An important advantage of the OSCE is the feasibility of assessing the validity of the exam. The most commonly used index for this is Pearson’s correlation, which is a useful tool for assessing the correlation between the OSCE score and the written exam and has been used in many published articles [ 17 – 19 ]. Most published reports have been about the advantages of OSCE as a reliable and valid examination method, but none have focused on the reliability of the indexes used in the assessment of the exam and whether a small difference between them means a single index is sufficient [ 17 , 20 ].…”

Section: Introductionmentioning

confidence: 99%

Is Cronbach’s alpha sufficient for assessing the reliability of the OSCE for an internal medicine course?

et al. 2015

View full text Add to dashboard Cite

BackgroundThe number of medical students accepted into medical programs is increasing, which has made the traditional long/short case style of examination difficult to conduct. At Dammam University, the program is shifting to the use of the Objective Structural Clinical Examination (OSCE), which may solve some of these difficulties, including issues with reliability, validity index and exam duration.ResultsA pilot study was conducted over one semester. A total of 207 examinees in three groups took the OSCE and written exams. The OSCE consisted of 18 clinical stations and required 3–4.3 h/day. The written exam contained 80 multiple-choice questions. The Cronbach’s alpha for each group was 0.7, 0.8, and 0.9. Correlations for all stations ranged from 0.7 to 0.8, which indicated good stability and internal consistency with minor differences in the progression of the indexes. The reliability of the written exam was 0.79, and the validity of the OSCE was 0.63, as assessed using Pearson’s correlation.ConclusionNo single reliability index can be considered a perfect assessment tool to solve this issue. Thus, at least two to three indexes should be used to ensure the reliability of the OSCE.

show abstract

Section: Introductionmentioning

confidence: 99%

Is Cronbach’s alpha sufficient for assessing the reliability of the OSCE for an internal medicine course?

et al. 2015

View full text Add to dashboard Cite

show abstract

“…These associations are consistent with the published reports of correlations between clinical assessments in medical schools and subtest scores on Step 2 CS. 4,5 However, our study went beyond previous work in an attempt to identify more specific performance metrics suggesting that a student might be at a higher risk of failing. Students with four or more very low faculty ratings of clinical performance in required clerkships were nearly 13 times more likely to fail than students with three or fewer low ratings.…”

Section: Discussionmentioning

confidence: 99%

“…However, their stepwise linear regression analysis revealed that preclinical grades and NBME Subject Examination scores in clerkships were more important predictors of Step 2 CS performance than were OSCE scores. 5 Neither study directly addressed the question of the student risk factors associated with failure.…”

mentioning

confidence: 99%

Identifying Students at Risk of Failing the USMLE Step 2 Clinical Skills Examination

Rosenthal

Russo

Berg

et al. 2019

Fam Med

View full text Add to dashboard Cite

Background and Objectives: New standards announced in 2017 could increase the failure rate for Step 2 Clinical Skills (CS). The purpose of this study was to identify student performance metrics associated with risk of failing. Methods: Data for 1,041 graduates of one medical school from 2014 through 2017 were analyzed, including 30 (2.9%) failures. Metrics included Medical College Admission Test, United States Medical Licensing Examination Step 1, and clerkship National Board of Medical Examiners (NBME) Subject Examination scores; faculty ratings in six clerkships; and scores on an objective structured clinical examination (OSCE). Bivariate statistics and regression were used to estimate risk of failing. Results: Those failing had lower Step 1 scores, NBME scores, faculty ratings, and OSCE scores (P<.02). Students with four or more low ratings were more likely to fail compared to those with fewer low ratings (relative risk [RR], 12.76, P<.0001). Logistic regression revealed other risks: low surgery NBME scores (RR 3.75, P=.02), low pediatrics NBME scores (RR 3.67, P=.02), low ratings in internal medicine (RR 3.42, P=.004), and low OSCE Communication/Interpersonal Skills (RR 2.55, P=.02). Conclusions: Certain medical student performance metrics are associated with risk of failing Step 2 CS. It is important to clarify these and advise students accordingly.

show abstract

“…Dong et al [7] found correlations between second year OSCEs with Step 2 CS (r = 0.25, p < 0.01) and with Step 3 (r = 0.26, p < 0.01). For third year OSCEs, the correlations also were significant with Step 2 CS (r = 0.27, p < 0.01) as they were with Step 3 (r = 0.15, p < 0.01).…”

Section: Introductionmentioning

confidence: 92%

Does Step 3 of the United States Medical Licensing Exam Measure Clinical Competence? A Predictive Validity Study

2016

View full text Add to dashboard Cite

Purpose The main purpose of the present study was to evaluate the predictive validity of medical school admissions data, preclinical and clinical achievement data, postgraduate evaluations, and the United States Medical Licensing Exam Step 1 and 2 data for performance on Step 3. Methods A total of 321 physicians (178 men, 55.4 %; 143 women, 44.6 %) participated. Admissions data, medical school preclinical achievement data, medical school clerkship clinical performance data, postgraduate clinical performance data, and USMLE Step 1 and 2 data were independent variables.Step 3 was the dependent variable in a backwards multiple regression. An analysis of covariance with Step 3 scores as the dependent variable, Step 2 scores as the covariate, and residency program (narrowly focused vs. broadly focused two levels) as the independent variable was conducted. Results The regression analyses resulted in an optimal model fit with the most parsimonious combination of independent variables (multiple R = 0.686, R 2 = 0.470, p < 0.001):Step 2 (β = 0.606) + verbal reasoning (β = 0.125) + PGY1 (β = 0.161) + Year 3 (β = 0.157). Candidates in broadly focused residencies scored higher onStep 3 compared to those in narrowly focused residencies (F = 5.17, p < 0.05). Conclusions The predictive validity of Step 2, verbal reasoning, PGY1 assessment, Year 3 average for Step 3 performance is supported by the optimal regression model. Candidates in broadly focused residencies scored higher onStep 3 compared to those in narrowly focused residencies. The results are evidence for the validity of Step 3 as a measure of clinical competence.

show abstract

Validity Evidence for Medical School OSCEs: Associations With USMLE^®Step Assessments

Abstract: These findings, although preliminary, provide some predictive validity evidence for the use of OSCEs in determining readiness of medical students for clinical practice and licensure.

Cited by 33 publications

References 13 publications

Is Cronbach’s alpha sufficient for assessing the reliability of the OSCE for an internal medicine course?

Is Cronbach’s alpha sufficient for assessing the reliability of the OSCE for an internal medicine course?

Identifying Students at Risk of Failing the USMLE Step 2 Clinical Skills Examination

Does Step 3 of the United States Medical Licensing Exam Measure Clinical Competence? A Predictive Validity Study

Contact Info

Product

Resources

About

Validity Evidence for Medical School OSCEs: Associations With USMLE®Step Assessments

Abstract: These findings, although preliminary, provide some predictive validity evidence for the use of OSCEs in determining readiness of medical students for clinical practice and licensure.

Cited by 33 publications

References 13 publications

Is Cronbach’s alpha sufficient for assessing the reliability of the OSCE for an internal medicine course?

Is Cronbach’s alpha sufficient for assessing the reliability of the OSCE for an internal medicine course?

Identifying Students at Risk of Failing the USMLE Step 2 Clinical Skills Examination

Does Step 3 of the United States Medical Licensing Exam Measure Clinical Competence? A Predictive Validity Study

Contact Info

Product

Resources

About

Validity Evidence for Medical School OSCEs: Associations With USMLE^®Step Assessments