Rater Training to Support High-Stakes Simulation-Based Assessments

Abstract

Competency-based assessment and an emphasis on obtaining higher-level outcomes that reflect physicians’ ability to demonstrate their skills has created a need for more advanced assessment practices. Simulation-based assessments provide medical education planners with tools to better evaluate the 6 Accreditation Council for Graduate Medical Education (ACGME) and American Board of Medical Specialties (ABMS) core competencies by affording physicians opportunities to demonstrate their skills within a standardized and replicable testing environment, thus filling a gap in the current state of assessment for regulating the practice of medicine. Observational performance assessments derived from simulated clinical tasks and scenarios enable stronger inferences about the skill level a physician may possess, but also introduce the potential of rater errors into the assessment process. This article reviews the use of simulation-based assessments for certification, credentialing, initial licensure, and relicensing decisions and describes rater training strategies that may be used to reduce rater errors, increase rating accuracy, and enhance the validity of simulation-based observational performance assessments.

Assessing team performance – Markers and methods

“…The main potential problems with making judgements about observed behaviours are those of validity (measuring what the tool purports to measure) and reliability (getting the same results each time). Reliability is necessary, but not sufficient, for validity, and it seems intuitive that trained raters and increasing experience are essential to achieve both [33]. This has obvious resource implications since assessors need to be recruited, trained and available.…”

Summative Simulated-Based Assessment in Nursing Programs

“…In the current authors' study of end-of-program assessment, a rater training program consistent with others reported in the literature was implemented (Eppich et al, 2015;Feldman et al, 2012). The goals of rater training were to ensure consistency across evaluators and by evaluators themselves in their observations of performance and ratings of those behaviors and to reach agreement about whether those observations represented competent performance of nursing students at the end of their program.…”

“…The quality of the ratings in a simulation-based assessment depends on the evaluators and the degree to which they accurately observe and evaluate the performance or skill (Feldman, Lazzara, Vanderbilt, & DiazGranados, 2012;Pangaro & Holmboe, 2008). Training should be conducted during multiple sessions in which evaluators have an opportunity to rate varied scenarios using the same tools they will use in the summative assessment, discuss their ratings and rationale with each other, and come to agreement regarding observed behaviors and competence.…”

“…With these multiple training sessions, combined with practice in scoring videos in between the face-to-face sessions, the overall ICC was 0.65, indicating good interrater reliability. In a high-stakes assessment, the evaluators need to accurately observe and rate performance (Feldman et al, 2012;Pangaro & Holmboe, 2008). Ratings should be consistent across evaluators.…”

An integrative framework for sensor-based measurement of teamwork in healthcare

“…[12][13][14] Observational measures incur labor costs of time spent observing, training, and monitoring raters over time. 15 This can be a large investment in effort, up to 20 h for some systems, 16 and largely limit observations to funded research. Although self-reporting and observation can serve critical roles in the training and periodic peer review processes, the frequency of feedback to clinicians likely remains low.…”