Judging quality through substantive conversations between markers

Abstract

Decisions by markers about quality in student work remain confusing to most students and markers. This may in part be due to the relatively subjective nature of what constitutes a quality response to an assessment task. This paper reports on an experiment that documented the process of decision-making by multiple markers at a university who assessed the same written student assessment responses. The paper analyses the professional conversations between those markers around their conceptions of quality in the student assessment responses. It was found that the markers appeared to share common understandings of quality in the context of the marking criteria and standards across the achievement levels awarded. However, despite these apparently shared notions of quality, in some cases different levels of achievement were awarded to the same student assessment responses. This suggests that that there is a clear need for explicitly stated standard descriptors for each level of achievement and that this must be interpreted through substantive professional conversations in the context of real student work.The key driver is the student work, and conversations amongst markers about what constitutes 'quality' in the context of the written and explicit criteria and standards of achievement that are available to students and markers alike are a necessity.

Evaluating the Quality of Higher Education Instructor-Constructed Multiple-Choice Tests: Impact on Student Grades

“…Grades are signals of achievement and show students their areas of strength and weakness and can inform instructors about the success of their teaching (Joughin, 2009;Yorke, 2009;Brown, 2010;Walvoord and Anderson, 2011). Obviously, the quality of assessment matters so that inferences and decisions by students and instructors, as well as external stakeholders (e.g., employers), can be made on a robust basis (Grainger et al, 2008).…”

Making judgements about students making work: Lecturers’ assessment practices in art and design

“…For example, Van der Shaff (2011), Orr (2007 and Grainger et al (2008) have all adopted this approach and a key benefit is its implicit naturalness. In the study reported here the conversations were happening anyway which contrasts with Grainger et al's study where the conversations are elicited by the researcher through the creation of an assessment scenario.…”

“…As a result we rejected the use of (quasi) experimental approaches where assessments are set up for the sole purpose of research. Grainger et al (2008) adopted a quasi experimental approach in a study where they asked multiple markers to assess the same pieces of student work. In their study the data consisted of the transcripts of the 'professional conversations' about the work.…”

External peer review of assessment: an effective approach to verifying standards?

“…Whilst unreliability in academics' assessment of student work is welldocumented, there is little prior data indicating how much specific criteria are interpreted differently although similar evidence was found by Grainger, Purnell and Zipf (2008). This is worthy of further investigation given the emphasis placed on analytical criteria in many qualification frameworks, lists of threshold outcomes and professional standards.…”