Do Individual Attendings’ Post-rotation Performance Ratings Detect Residents’ Clinical Performance Deficiencies?

Abstract

A large percentage of performance deficiencies only became apparent when the attending physicians discussed performance at the annual evaluation meetings. Annual evaluation meetings may (1) make patterns of residents' behavior apparent that were not previously apparent to individual physicians, (2) provide evidence that strengthens the individual attending's preexisting convictions about residents' performance deficiencies, or (3) lead to erroneous conclusions. The authors believe deficiencies were real and that their findings can be explained by a combination of reasons one and two.

Methods and Outcomes for the Remediation of Clinical Reasoning

“…[1][2][3][4][5] Most learners have difficulty with cognitive domains, including the application of medical knowledge in the form of clinical reasoning. [3][4][5][6][7][8][9] There is a lack of evidence to guide best practices in remediation and no widely accepted framework for the remediation of clinical reasoning. When focus groups were conducted with clinical educators, the participants stated they did not have a structured or systematic approach for remediation, and expressed skepticism about the impact of remediation on clinical reasoning.…”

Group Assessments of Resident Physicians Improve Reliability and Decrease Halo Error

“…Schwind and colleagues found that discussion at a surgical resident evaluation committee, compared to individual assessments, identified an increased number of deficiencies across three performance domains. 5 By comparison, our study provides evidence of significant differences in reliability and a halo effect between the two methods of assessment. Future studies should examine inpatient group assessment for resident Figure 2.…”

“…Individual faculty members have only brief contact and limited exposure to resident behaviors, so they may miss significant deficiencies in performance. 5 Additionally, faculty members often provide minimal feedback 21 and may place variable emphasis on different performance criteria. 7,10 We report an integrative faculty group assessment model for internal medicine continuity clinics that uses pooled faculty observations to identify learner deficiencies, enable more specific feedback, and provide ongoing case-based faculty development.…”

“…4 For example, unique post-rotation performance ratings often neglect a large percentage of deficiencies apparent at summative, end-of-year progress judgments. 5 Ratings forms are sometimes inaccurate and provide little specific formative feedback to residents. 6 It is difficult to assure the consistent application of performance standards across a large number of faculty members.…”

A Faculty Development Program to Reduce Rater Error on Milestone-Based Assessments

“…Studies have shown that end-of-rotation evaluations completed by faculty lack reliability and fail to identify important deficiencies in residents' performance. 1,2 A number of factors account for the less-than-acceptable reliability, including variations in clinical context or patient complexity, 3 lack of longitudinal experiences with residents and patients, 4 and rater error. Types of rater error include leniency error, in which residents' abilities are rated higher than performance merits; halo or reverse halo, when performance in 1 domain affects ratings in other domains; range restriction, in which ratings fall within a narrow range; and error of undifferentiation, in which raters fail to differentiate among different performance domains.…”