Reconceptualizing Variable Rater Assessments as Both an Educational and Clinical Care Problem

Abstract

The public is calling for the U.S. health care and medical education system to be accountable for ensuring high-quality, safe, effective, patient-centered care. As medical education shifts to a competency-based training paradigm, clinician educators' assessment of and feedback to trainees about their developing clinical skills becomes paramount. However, there is substantial variability in the accuracy, reliability, and validity of the assessments faculty make when they directly observe trainees with patients. These difficulties have been treated primarily as a rater cognition problem focusing on the inability of the assessor to make reliable and valid assessments of the trainee.The authors' purpose is to reconceptualize the rater cognition problem as both an educational and clinical care problem. The variable quality of faculty assessments is not just a psychometric predicament but also an issue that has implications for decisions regarding trainee supervision and the delivery of quality patient care. The authors suggest that the frame of reference for rating performance during workplace-based assessments be the ability to provide safe, effective, patient-centered care. The authors developed the Accountable Assessment for Quality Care and Supervision equation to remind faculty that supervision is a dynamic, complex process essential for patients to receive high-quality care. This fundamental shift in how assessment is conceptualized requires new models of faculty development and emphasizes the essential and irreplaceable importance of the clinician educator in trainee assessment.

Entrustment Decision Making in Clinical Training

“…57 Clinicians who assess competencies require a mind-set somewhat remote from their daily mode of thinking. Improving the alignment of rating tasks with regular clinical tasks is likely to increase their quality, 8,58,59 such as relabeling abstract rating scales (e.g., poor to excellent) with the question "How much supervision is justified?" 9…”

“…Second, making safe, effective, patient-centered care the frame of reference for the entrustment brings the focus back more firmly to the patient. 59 Third, explicit entrustment requires the hard conversations around curriculum design.…”

Milestones: Not Millstones but Stepping Stones

“…The intended outcome of training-safe, effective, and patientcentered care-should be included in the assessment of the learner. 4 There are 2 additional benefits to those described by Norman et al 1 5. Providing better, more accurate assessments in the messy, real world contexts at the point of care: Fair and accurate assessment using direct observation is dependent on (1) a shared mental model of what behaviors to look for (Milestones); and (2) faculty development to ''sharpen the people rather than the instruments.''…”

Advancing Competency-Based Medical Education: A Charter for Clinician–Educators

“…The tools that we use should be part of a structured program of assessment 37 and be guided by their "utility," which is defined by van der Vleuten and Schuwirth 32 as the "multiplicative product of their reliability, validity, cost-effectiveness, feasibility, and educational impact." Kogan and colleagues 38 have recently recommended that patient outcomes should also inform our assessment of learners.…”