Examiner training: A study of examiners making sense of norm-referenced feedback

Abstract

Examiner training has an inconsistent impact on subsequent performance. To understand this variation, we explored how examiners think about changing the way they assess. Method We provided comparative data to seventeen experienced examiners about their assessments, captured their sense-making processes using a modified think-aloud protocol, and identified patterns by inductive thematic analysis. Results We observed five sense-making processes: (1) testing personal relevance (2) interpretation (3) attribution (4) considering the need for change, and (5) considering the nature of change. Three observed meta-themes describe the manner of examiners' thinking: Guarded curiosity-where examiners expressed curiosity over how their judgements compared with others', but they also expressed guardedness about the relevance of the comparisons. Dysfunctional assimilation-where examiners' interpretation and attribution exhibited cognitive anchoring, personalisation, and affective bias. Moderated conservatism-where examiners expressed openness to change, but also loyalty to their judgement-framing values and aphorisms. Conclusions Our examiners engaged in complex processes as they considered changing their assessments. The 'stabilising' mechanisms some used resembled learners assimilating educational feedback. If these are typical examiner responses, they may well explain the variable impact of examiner training, and have significant implications for the pursuit of meaningful and defensible judgement-based assessment.

Re-conceptualising and accounting for examiner (cut-score) stringency in a ‘high frequency, small cohort’ performance test

“…In terms of alleviating the problem of excessive variation in cut-score stringency, the literature suggests that feedback to examiners on their judgments can sometimes help to reduce (score) stringency (Wong et al 2020 ), whilst recognising that this is a complex area and is not always effective (Crossley et al 2019 ; Gingerich et al 2011 ).The linear mixed modelling automatically produces a measure of cut-score stringency for each examiner, and this could form part of feedback to them of their performance relative to their peers. This information would have to be carefully mediated as it might be difficult for examiners to interpret or act on it compared to the more conventional feedback on scores.…”

Interviewer judgment-making as a social practice: Insights from an institutional observational study in postgraduate selection

“…[38] [39] [31] In the broader assessment arena, there is a general skepticism that further training of judges makes much impact on reducing these systematic sources of error in their interviewing performance, [24,40] although some approaches for giving speci c feedback back to examiners are thought to have promise. [41,42] Study Aims and research questions.…”