Re-conceptualising and accounting for examiner (cut-score) stringency in a ‘high frequency, small cohort’ performance test

Abstract

Variation in examiner stringency is an ongoing problem in many performance settings such as in OSCEs, and usually is conceptualised and measured based on scores/grades examiners award. Under borderline regression, the standard within a station is set using checklist/ domain scores and global grades acting in combination. This complexity requires a more nuanced view of what stringency might mean when considering sources of variation of cut-scores in stations. This study uses data from 349 administrations of an 18-station, 36 candidate single circuit OSCE for international medical graduates wanting to practice in the UK (PLAB2). The station-level data was gathered over a 34-month period up to July 2019. Linear mixed models are used to estimate and then separate out examiner (n = 547), station (n = 330) and examination (n = 349) effects on borderline regression cut-scores. Examiners are the largest source of variation in cut-scores accounting for 56% of variance in cut-scores, compared to 6% for stations, < 1% for exam and 37% residual. Aggregating to the exam level tends to ameliorate this effect. For 96% of examinations, a 'fair' cutscore, equalising out variation in examiner stringency that candidates experience, is within one standard error of measurement (SEM) of the actual cut-score. The addition of the SEM to produce the final pass mark generally ensures the public is protected from almost all false positives in the examination caused by examiner cut-score stringency acting in candidates' favour.

Pass/fail decisions and standards: the impact of differential examiner stringency on OSCE outcomes

“…This study confirms that examiner stringency is a very important influence on stationlevel scoring/grading (Tables 4 and 7), and that adjusting for this does impact on stationlevel scores (Table 6). These findings are consistent with a wide range of literature (Homer, 2020;McManus et al, 2006;Santen et al, 2021;Yeates et al, 2018Yeates et al, , 2021, but our work suggests that acceptable levels of overall assessment reliability can be achieved provided the number of stations is large enough (Table 5)-again consistent with other empirical and/or psychometric work (Bloch & Norman, 2012;Park, 2019). There is a lot of residual variance at the station level, and these results do suggest, however, that a focus on examlevel, rather than station-level, performance of a candidates is likely to be more meaningful in terms of good decision-making.…”

“…We argue that the statistical methods used here are valuable in quantifying error and its impact on the exam overall, but can never be truly confident that all sources of error have been captured and accounted for properly. This in turn implies that adjusting candidatelevel scores and using these for actual decision-making is hard to justify, as has been argued elsewhere (Homer, 2020).…”

“…Analysis of station-level data in this setting (i.e. all candidate level data aggregated to station level) indicates that examiner variation in station-level standards can be accounted for by appropriate use of the standard error of measurement (Hays et al, 2008;Homer, 2020). This paper provides a more finegrained analysis that includes candidate effects using fully anonymised candidate-level data from the same setting.…”

“…The stated strength of the OSCE, however, is that such error might largely balance out over the exam as a whole through wide sampling of examiners and stations, and via attempts to standardise the overall assessment process through, for example, structured scoring instruments, and appropriate examiner training (Harden et al, 2015, Chapters 9 and 11). However, some research brings into question the assumption that scoring error largely cancels out at the exam level (Homer, 2020;Yeates et al, 2018Yeates et al, , 2021.…”

Determining influence, interaction and causality of contrast and sequence effects in objective structured clinical exams

“…As DRIFT effects might result in additional station fails for some students, this could produce unwarranted failure for some candidates. If determined to be of sufficient importance in some instances, this effect could be mitigated by either adjusting students' station-level scores or the station-level pass mark 43 exams over a programme. Importantly, the small (and inconsistently observed) magnitude of the effect we have found in this study may be considered insufficiently important to warrant alterations of this nature, given that other effects (such as the number of OSCE stations 44 ) are known to have a greater influence on reliability of the test.…”