Developing a video‐based method to compare and adjust examiner effects in fully nested OSCEs

Abstract

Background Although averaging across multiple examiners’ judgements reduces unwanted overall score variability in objective structured clinical examinations ( OSCE ), designs involving several parallel circuits of the OSCE require that different examiner cohorts collectively judge performances to the same standard in order to avoid bias. Prior research suggests the potential for important examiner‐cohort effects in distributed or national examinations that could compromise fairness or patient safety, but despite their importance, these effects are rarely investigated because fully nested assessment designs make them very difficult to study. We describe initial use of a new method to measure and adjust for examiner‐cohort effects on students’ scores. Methods We developed video‐based examiner score comparison and adjustment ( VESCA ): volunteer students were filmed ‘live’ on 10 out of 12 OSCE stations. Following the examination, examiners additionally scored station‐specific common‐comparator videos, producing partial crossing between examiner cohorts. Many‐facet Rasch modelling and linear mixed modelling were used to estimate and adjust for examiner‐cohort effects on students’ scores. Results After accounting for students’ ability, examiner cohorts differed substantially in their stringency or leniency (maximal global score difference of 0.47 out of 7.0 [Cohen's d = 0.96]; maximal total percentage score difference of 5.7% [Cohen's d = 1.06] for the same student ability by different examiner cohorts). Corresponding adjustment of students’ global and total percentage scores altered the theoretical classification of 6.0% of students for both measures (either pass to fail or fail to pass), whereas 8.6–9.5% students’ scores were altered by at least 0.5 standard deviations of student ability. Conclusions Despite typical reliability, the examiner cohort that students encountered had a potentially important influence on their score, emphasising the need for adequate sampling and examiner training. Development and validation of VESCA may offer a means to measure and adjust for potential systematic differences in scoring patterns that could exist between locations in distributed or national OSCE examinations, thereby ensuring equivalence and fairness.

Re-conceptualising and accounting for examiner (cut-score) stringency in a ‘high frequency, small cohort’ performance test

“…3 Distribution of differences between unadjusted and fair exam-level cut-scores (%) (n = 349) clinical performance (Harden et al 2015, Chapter 3;Khan et al 2013b). This finding also complements the research on the importance of examiner effects on scoring in OSCEs (McManus et al 2006;Yeates et al 2018;Yeates and Sebok-Syer 2017).…”

Setting defensible standards in small cohort OSCEs: Understanding better when borderline regression can ‘work’

“…In one sense, having a single assessor is more likely to bring consistency to the marking of the station since all candidates will be observed by the same set of assessors across the exam. On the other hand, in small cohorts single assessors are confounded with stations and there is no easy way to directly compare assessor stringency at the station level , Yeates et al 2018. The presence of a single assessor per station (rather than multiple across circuits) may also contribute to a lack of calibration compared to a group of peers all examining the same station.…”

Understanding and developing procedures for video-based assessment in medical education

“…Researchers filmed the simulated OSCE stations using a variety of filming methods. This included fixed, wide angle ceiling cameras (identical to filming in Yeates et al (2019); tripod based camcorders, positioned in various places within the room, and using varied degrees of zoom; head cameras worn by the examiner; and wall mounted pan/tilt/ zoom CCTV cameras positioned in various places within the room (see Figure 1). Sound was collected variously using ceiling hanging microphones; focused microphones placed on the camcorders; and table-top condenser microphones.…”

“…Common to all of these approaches is the assumption that assessors' video-based judgements are equivalent to their judgements of live performances. Several studies have shown equivalent scores between live and video-based modalities (Ryan et al 1995;Vivekananda-Schmidt et al 2007;Chen et al 2019;Yeates et al 2019). This observation is not, however, universal.…”

“…In recent years, several innovations have attempted to address perennial limitations in the consistency (Eva 2018) and educational impact (Harrison et al 2017) of assessment in medical education by using video technology to: enhance quality assurance of examiners' scoring (Yeates et al 2019); support assessor training through benchmarking (McManus and Omer 2017), enable remote examining (Vivekananda-Schmidt et al 2007;Chen et al 2019), or provide a detailed review of students' performance within feedback conversations (Eeckhout et al 2016). Whilst these approaches offer significant promise, the implications of using video-based judgements within assessment has received comparatively little scrutiny.…”