The inter-rater reliability and internal consistency of a clinical evaluation exercise

Kroboth, Frank J.; Hanusa, Barbara H.; Parker, Susan C.; Coulehan, John L.; Kapoor, Wishwa N.; Brown, F.; Karpf, Michael; Levey, Gerald S.

doi:10.1007/bf02598008

Cited by 73 publications

(53 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…The amount of rater experience did not change the results. Similar results have been reported by Kreiter and colleagues 16 (more than 8 ratings needed), Kwolek et al 31 (7-8 ratings), Ramsey et al 32 (11 ratings), Violato et al 33 (10 ratings), and Kroboth et al 34 (6-10 ratings). All reports agree that somewhere between 7 and 11 ratings are necessary to achieve a generalizable global estimate of competence when raters are basing ratings on a nonsystematic sample of observations.…”

Section: Accuracy In Estimating Competence (Estimating From a Discretsupporting

confidence: 87%

SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings

Williams¹,

Klamen²,

McGaghie³

2003

Teaching and Learning in Medicine

351

250

View full text Add to dashboard Cite

show abstract

Section: Accuracy In Estimating Competence (Estimating From a Discretsupporting

confidence: 87%

SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings

Williams¹,

Klamen²,

McGaghie³

2003

Teaching and Learning in Medicine

351

250

View full text Add to dashboard Cite

show abstract

“…Multiple methods exist for clinical competency evaluation, including the housestaff evaluation forms provided by respective boards, [1][2][3][4] the In-Training Examination (ITE), 5 medical record review, 6 credentialling for procedures, the Objectively Structured Clinical Exam, 7 nurse evaluations, 8 patient satisfaction survey information, [9][10][11] housestaff self-assessment, 12 and the Clinical Evaluation Exercise (CEX). [13][14] Medical knowledge, as measured by the American Board of Internal Medicine (ABIM) Certifying Examination, correlates well with achievement on the ABIM housestaff evaluation form and ITE. 15 The ABIM housestaff evaluation form may be valid for assessing overall clinical competence, but it is less useful for providing feedback in specific areas to individual residents.…”

mentioning

confidence: 99%

The Impact of Feedback to Medical Housestaff on Chart Documentation and Quality of Care in the Outpatient Setting

Opila

1997

J Gen Intern Med

View full text Add to dashboard Cite

OBJECTIVE:To determine whether feedback from attending physicians to residents about outpatient medical records improves chart documentation and quality of care. DESIGN:Cross-sectional study with repeated measures.SETTING: Primary care internal medicine clinic at a metropolitan community hospital. PATIENT/PARTICIPANTS: Fifteen interns and 20 residents. INTERVENTION:Attending physicians reviewed at least two charts for each resident on three occasions about 4 months apart and then discussed their findings with the residents. MEASUREMENTS AND MAIN RESULTS:Explicit criteria defined the extent of chart documentation and the comprehensiveness of care delivery. Attending physicians also made a subjective assessment of the overall quality of care. All results were converted to 0-to-1 scales. From the first to the third period, chart documentation increased from 0.60 to 0.86 ( p Ͻ .001), but there were no significant changes in the delivery of care or in the subjective assessments of the overall quality of care. nternal medicine resident training programs are required to evaluate the competence of their trainees in the areas of medical knowledge, clinical competence, attitudes, and behaviors. Multiple methods exist for clinical competency evaluation, including the housestaff evaluation forms provided by respective boards, 1-4 the In-Training Examination (ITE), 5 medical record review, 6 credentialling for procedures, the Objectively Structured Clinical Exam, 7 nurse evaluations, 8 patient satisfaction survey information, 9-11 housestaff self-assessment, 12 and the Clinical Evaluation Exercise (CEX). [13][14] Medical knowledge, as measured by the American Board of Internal Medicine (ABIM) Certifying Examination, correlates well with achievement on the ABIM housestaff evaluation form and ITE. 15 The ABIM housestaff evaluation form may be valid for assessing overall clinical competence, but it is less useful for providing feedback in specific areas to individual residents. 2 The ABIM housestaff evaluation form, ITE, CEX, and faculty predictions of completeness, however, are not even moderately correlated with residents' performance of components of the physical examination. 6 Program directors are challenged, therefore, to be innovative in finding reliable and valid methods for evaluating the clinical competence of housestaff and providing opportunities for improvement if deficiencies are identified. CONCLUSIONS:Medical record review offers an attractive mechanism for evaluating clinical competence because of its ease of implementation relative to other methods such as standardized patients. The medical record documents the specific components of patient care and also demonstrates physician thought processes and outcomes of patient management. Feedback about performance to resident physicians should lead to improvement in clinical competence.The current practice environment of managed care involves many efforts to improve quality utilizing office record review. For example, the National Council for Quality Assurance has de...

show abstract

“…[1][2][3] The mini-CEX was designed to assess residents with respect to clinical skills, attitudes, and behaviors that are essential in providing high-quality patient care. In a study of 421 PGY-1 IM residents from 21 programs, evaluated by 316 physicians, the 7 components of competence were highly correlated and reliable.…”

mentioning

confidence: 99%

Education Research: Bias and poor interrater reliability in evaluating the neurology clinical skills examination

Schuh

London²,

Neel

et al. 2009

Neurology

View full text Add to dashboard Cite

Objective: The American Board of Psychiatry and Neurology (ABPN) has recently replaced the traditional, centralized oral examination with the locally administered Neurology Clinical Skills Examination (NEX). The ABPN postulated the experience with the NEX would be similar to the Mini-Clinical Evaluation Exercise, a reliable and valid assessment tool. The reliability and validity of the NEX has not been established.Methods: NEX encounters were videotaped at 4 neurology programs. Local faculty and ABPN examiners graded the encounters using 2 different evaluation forms: an ABPN form and one with a contracted rating scale. Some NEX encounters were purposely failed by residents. Cohen's kappa and intraclass correlation coefficients (ICC) were calculated for local vs ABPN examiners.Results: Ninety-eight videotaped NEX encounters of 32 residents were evaluated by 20 local faculty evaluators and 18 ABPN examiners. The interrater reliability for a determination of pass vs fail for each encounter was poor (kappa 0.32; 95% confidence interval [CI] ϭ 0.11, 0.53). ICC between local faculty and ABPN examiners for each performance rating on the ABPN NEX form was poor to moderate (ICC range 0.14 -0.44), and did not improve with the contracted rating form (ICC range 0.09 -0.36). ABPN examiners were more likely than local examiners to fail residents. Conclusions:There is poor interrater reliability between local faculty and American Board of Psychiatry and Neurology examiners. A bias was detected for favorable assessment locally, which is concerning for the validity of the examination. Further study is needed to assess whether training can improve interrater reliability and offset bias. The American Board of Psychiatry and Neurology (ABPN) eliminated its centralized oral examination for several reasons. First, the encounters were not standardized. Second, a single high-stakes, high-stress encounter is not necessarily representative of a candidate's performance. Third, deficiencies were difficult to remediate for failed candidates as they had already completed training. To replace the centralized oral examination, the ABPN Neurology Council voted to require residency programs to administer a set of clinical skills examinations called the Neurology Evaluation Exercise (NEX) to all residents entering training as of July 1, 2005, and document every graduating resident's satisfactory performance on these. The ABPN proposed this approach after reviewing the American Board of Internal Medicine (ABIM) experience with clinical skills evaluation in internal medicine (IM) residency using the Mini-Clinical

show abstract

The inter-rater reliability and internal consistency of a clinical evaluation exercise

Abstract: Although the CEX is a valuable didactic tool, its psychometric properties preclude reliable assessment of clinical skills as a one-time observation.

Cited by 73 publications

References 23 publications

SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings

SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings

The Impact of Feedback to Medical Housestaff on Chart Documentation and Quality of Care in the Outpatient Setting

Education Research: Bias and poor interrater reliability in evaluating the neurology clinical skills examination

Contact Info

Product

Resources

About