What makes AS marking reliable? An experiment with some stages from the standardisation process

Abstract

It is particularly important that GCSE and A-level marking is valid and reliable as it affects the life chances of many young people in England. Current developments in marking technology are coinciding with potential changes in procedures to ensure valid and reliable marking. In this research the effectiveness of procedures to facilitate the reliability of marking were evaluated using a design with three interventions. Two of the interventions, the standardisation meeting and personal feedback, are similar to the conventional approach in GCSE and A-level practice but within the constraints of a research study. The other intervention, pre-written feedback, is how examples of marked scripts might be presented for part of examiner standardisation or training in other situations; it is also similar to the approach to feedback used in earlier research by Shaw and Baird et al. Each participating examiner experienced two of the three types of intervention. Marking data were gathered before and after the first intervention as well as after the second intervention. Inter-examiner reliability was analysed using ANOVA models. Due to the different levels of inter-examiner reliability at the beginning of the experiment it was hard to compare the effectiveness of the different interventions. Nevertheless, this research highlights that: (1) in line with previous research, standardisation meetings on their own are not particularly effective for experienced examiners;(2) as has been found in previous research, more experienced examiners were more lenient; (3) most combinations of interventions (combinations of standardisation processes -including the conventional approach) led to a decline in the size of the marking error; and (4) sometimes feedback (personal or pre-written) was effective and sometimes it was not.

Examiner training: A study of examiners making sense of norm-referenced feedback

“…This even holds true for feedback-driven training. In advanced secondary education in the UK, only a sub-group of examiners becomes more consistent at scoring following such feedback (Greatorex and Bell 2008). In a spoken English language assessment, examiners valued the feedback but there was no impact on their behaviour (Knoch 2011).…”

Feedback effectiveness in professional learning contexts

“…These details include consistent, transparent model answers or exemplars and annotated comments on models (Li & De Luca, ). In professional examination contexts, the articulation of marking rationales to examiners in feedback has been found to be effective (Greatorex & Bell, ). This point also relates to the work of Knoch et al .…”

“…Orientation is receptivity to feedback and describes the likeliness of a learner to seek, value and process feedback. Greatorex and Bell () found some examiners were more apt to attend to feedback than others. This is also mirrored in the management literature where Linderbaum and Levy () note that recipients differ in their responses to feedback.…”

“…There are some notable exceptions to this (e.g. Shaw, 2002;Baird et al, 2004;Meadows & Billington, 2005;Greatorex & Bell, 2008;Chamberlain & Taylor, 2011;May, 2013), but none of these look at the nature of feedback in the professional examiner context.…”

Articulation Work: Insights into examiners' expertise from their remote feedback interactions

“…4 discourse. Research evidence shows that examiner standardisation and feedback processes can help to align examiners' judgements with each other, with feedback given to examiners by more senior examiners reducing the scale of marking differences between them (Shaw, 2002;Meadows & Billington, 2005;Greatorex & Bell, 2008).…”