Education Research: Bias and poor interrater reliability in evaluating the neurology clinical skills examination

Abstract

Objective: The American Board of Psychiatry and Neurology (ABPN) has recently replaced the traditional, centralized oral examination with the locally administered Neurology Clinical Skills Examination (NEX). The ABPN postulated the experience with the NEX would be similar to the Mini-Clinical Evaluation Exercise, a reliable and valid assessment tool. The reliability and validity of the NEX has not been established.Methods: NEX encounters were videotaped at 4 neurology programs. Local faculty and ABPN examiners graded the encounters using 2 different evaluation forms: an ABPN form and one with a contracted rating scale. Some NEX encounters were purposely failed by residents. Cohen's kappa and intraclass correlation coefficients (ICC) were calculated for local vs ABPN examiners.Results: Ninety-eight videotaped NEX encounters of 32 residents were evaluated by 20 local faculty evaluators and 18 ABPN examiners. The interrater reliability for a determination of pass vs fail for each encounter was poor (kappa 0.32; 95% confidence interval [CI] ϭ 0.11, 0.53). ICC between local faculty and ABPN examiners for each performance rating on the ABPN NEX form was poor to moderate (ICC range 0.14 -0.44), and did not improve with the contracted rating form (ICC range 0.09 -0.36). ABPN examiners were more likely than local examiners to fail residents. Conclusions:There is poor interrater reliability between local faculty and American Board of Psychiatry and Neurology examiners. A bias was detected for favorable assessment locally, which is concerning for the validity of the examination. Further study is needed to assess whether training can improve interrater reliability and offset bias. The American Board of Psychiatry and Neurology (ABPN) eliminated its centralized oral examination for several reasons. First, the encounters were not standardized. Second, a single high-stakes, high-stress encounter is not necessarily representative of a candidate's performance. Third, deficiencies were difficult to remediate for failed candidates as they had already completed training. To replace the centralized oral examination, the ABPN Neurology Council voted to require residency programs to administer a set of clinical skills examinations called the Neurology Evaluation Exercise (NEX) to all residents entering training as of July 1, 2005, and document every graduating resident's satisfactory performance on these. The ABPN proposed this approach after reviewing the American Board of Internal Medicine (ABIM) experience with clinical skills evaluation in internal medicine (IM) residency using the Mini-Clinical

Reconceptualizing Variable Rater Assessments as Both an Educational and Clinical Care Problem

“…[20][21][22] The implication of these errors and biases is that different raters rate the same encounter differently 5,23 and make different pass/fail decisions. 24 The etiology of high interrater variability is likely multifactorial. Multiple complex social and psychological processes are involved when faculty make judgments about a trainee.…”

Neurology objective structured clinical examination reliability using generalizability theory

“…Finally, within the field of neurology, reliability has been examined both in educational endeavors 35 and clinical applications. [36][37][38] While OSCEs are a common example of a multifaceted assessment, many if not all applications of assessment involve multiple facets.…”

Education Research: Unsatisfactory NEX rating correlations

“…1 Of 98 total encounters, the ABPN examiners assigned a failing grade to 27, but local faculty recommended a failing grade for only 11 (40.7%) of these. Local faculty were twice as likely to agree with the ABPN examiners who assigned a passing grade to a given encounter than those who assigned a failing grade.…”