Measurement precision for repeat examinees on a standardized patient examination

Abstract

Examinees who initially fail and later repeat an SP-based clinical skills exam typically exhibit large score gains on their second attempt, suggesting the possibility that examinees were not well measured on one of those attempts. This study evaluates score precision for examinees who repeated an SP-based clinical skills test administered as part of the US Medical Licensing Examination sequence. Generalizability theory was used as the basis for computing conditional standard errors of measurement (SEM) for individual examinees. Conditional SEMs were computed for approximately 60,000 single-take examinees and 5,000 repeat examinees who completed the Step 2 Clinical Skills Examination(®) between 2007 and 2009. The study focused exclusively on ratings of communication and interpersonal skills. Conditional SEMs for single-take and repeat examinees were nearly indistinguishable across most of the score scale. US graduates and IMGs were measured with equal levels of precision at all score levels, as were examinees with differing levels of skill speaking English. There was no evidence that examinees with the largest score changes were measured poorly on either their first or second attempt. The large score increases for repeat examinees on this SP-based exam probably cannot be attributed to unexpectedly large errors of measurement.

Hawks, Doves and Rasch decisions: Understanding the influence of different cycles of an OSCE on students’ scores using Many Facet Rasch Modeling

“…Generalisability analysis works best when examiners are fully "crossed" with students and stations (i.e., all examiners observe all students at all stations); however, the model can produce useful data when some partial crossing is available (Shavelson, Webb, & Rowley, 1989). With a fully nested design (as typically observed in most medical schools), generalizability is more limited, as it is not able to disentangle variance due to examiner variability from that due to students' abilities (Raymond, Swygert, & Kahraman, 2012). Nesting occurs when, for example, students are examined at each station by just one examiner.…”

Resitting a high-stakes postgraduate medical examination on multiple occasions: nonlinear multilevel modelling of performance in the MRCP(UK) examinations

“…Raymond et al [ 6 ] reported that there were large differences in the internal factor structure of marks from candidates passing or failing a clinical assessment at their first attempt but at a second attempt when they passed, the candidates had a factor structure similar to those passing at their first attempt. An additional paper from the same group has also demonstrated that measurement error is equivalent at first and second attempts [ 8 ]. Taken together, those findings do not constitute what the GMC referred to as a 'very clear rationale' on which policy might be implemented.…”