Sources of variability in medical student evaluations on the internal medicine clinical rotation

Abstract

Objectives: To explore the sources of variability in evaluator ratings among third year medical students in the Internal Medicine clinical rotation. Also, to examine systematic effects and variability introduced by differences in the various student, evaluator, and evaluation settings. Methods: A multilevel model was used to estimate the amount of between-student, between-rater and raterstudent interaction variability present in the students' clinical evaluations in a third year internal medicine clinical rotation. Within this model, linear regression analysis was used to estimate the effect of variables on the students' numerical evaluation scores and the reliability of those scores. Results: A total of 2,747 evaluation surveys were collected from 389 evaluators on 373 students over 4.5 years. All surveys used a nine-point grading scale, and therefore all results are reported on this scale. The calculated betweenrater, between-student and rater-student interaction variance components were 0.50, 0.27 and 0.62, respectively. African American/Black students had lower scores than Caucasian students by 0.58 points (t=-3.28; P=0.001). No gender effects were noted. Conclusions: These between-rater and between-student variance components imply that the evaluator plays a larger role in the students' scores than the students themselves. The residual rater-student interaction variance was larger and did not change by accounting for the measured demographic variables. This implies there is significant variability in each rater-student interaction that remains unexplained. This could contribute to unreliability in the system, requiring that students receive between 8 and 17 clinical evaluations to achieve 80% reliability.

Academic performance of fnal year medical students at kerbala medical college

“…24 No gender difference was found in achievements in all parts of the final year examination in Kerbala Medical College and this was similar to the findings in many studies in other countries. [25][26][27] However, female students were consistently found in the literature to perform better than males in their medical training which might be related to higher motivation. A systematic review by Ferguson and his colleagues reported that a growing body of research explores whether different motivational, academic, and demographic factors influence the performance of male and female women where motivation seems to be important.…”

“…25 Multivariate regression analysis approach to studying predictors of success in medical training was suggested in a systematic review of about twenty two thousands medical students (21) . Predictors are likely to be inter-correlated, as are outcome measures.…”

The Effect of Pass/Fail Exam Grading on Exam Performance in a Pediatric Clerkship

“…In a study of internal medicine clerkship evaluations, there was greater variance in the scoring between evaluators than in scores between students. 4 This implies that variance in clinical performance scores depends more heavily on the evaluator than truly on student performance.…”