The problem of assessing problem solving: can comparative judgement help?

Abstract

School mathematics examination papers are typically dominated by short, structured items that fail to assess sustained reasoning or problem solving. A contributory factor to this situation is the need for student work to be marked reliably by a large number of markers of varied experience and competence. We report a study that tested an alternative approach to assessment, called comparative judgement, which may represent a superior method for assessing open-ended questions that encourage a range of unpredictable responses. An innovative problem solving examination paper was specially designed by examiners, evaluated by mathematics teachers, and administered to 750 secondary school students of varied mathematical achievement. The students' work was then assessed by mathematics education experts using comparative judgement as well as a specially designed, resourceintensive marking procedure. We report two main findings from the research. First, the examination paper writers, when freed from the traditional constraint of producing a mark scheme, designed questions that were less structured and more problem-based than is typical in current school mathematics examination papers. Second, the comparative judgement approach to assessing the student work proved successful by our measures of inter-rater reliability and validity. These findings open new avenues for how school mathematics, and indeed other areas of the curriculum, might be assessed in the future.Keywords: assessment, problem solving, validity, comparative judgement 3 Typical mathematics exams are not fit for the purpose of assessing students' mathematical knowledge and skills. Analyses of the content and style of examination papers support the conjecture that mathematics examination papers comprise mainly short items that assess the rote learning of isolated facts and procedures (Berube, 2004; NCETM, 2009; Noyes, Wake, Drake & Murphy, 2011). An example question from a recent General Certificate for Secondary Education (GCSE) examination paper, a national qualification in England taken by most school leavers, illustrates the problem, as shown in Figure 1. ***FIGURE 1 ABOUT HERE*** At first glance, the question looks promising for assessing students' mathematical knowledge and skills. It makes use of a calendar context thereby appealing to the everyday relevance of mathematics. It also builds on students' experience of a counting system grouped in 7s to introduce an interesting generality that wherever the 2 by 2 square is positioned the provided algorithm will always give 7. However, to achieve full marks, all a student needs to do is compute the provided algorithm using the provided inputs. No explanation or proof of why the result is always 7 is required or rewarded. An efficient examination taker can achieve full marks without noticing there is a mathematically interesting generality at all.The question might be improved by asking students to compute the algorithm for a few 2 by 2 squares of their own choosing, and then asking them to explain what they ...

When teachers compare argumentative texts: Decisions informed by multiple complex aspects of text quality

“…Although the latter were mentioned in almost half of the decision statements, this was almost always accompanied with a reflection on the complex aspects. Based on this study we can assume that comparative methods in particular are a valid method for the assessment of complex skills, as the comparative methods enable the teacher to obtain reliable scores for complex skills more easily than using analytic methods (Coertjens, Lesterhuis, Verhavert, Van Gasse, & De Maeyer, 2017;Jones & Inglis, 2015) and teachers focus on these higher order skills while assessing the texts. This also means that when rule-applying skills need to be assessed, other scoring methods such as rubrics or thick boxes might be more suitable.…”

Measuring conceptual understanding in randomised controlled trials: Can comparative judgement help?

“…Researchers working in subjects other than mathematics need to consider carefully which population to draw judges from. For the case of mathematics we have previously found that educationalists, research students and research-active mathematicians all produce valid and reliable scores when judging students' work (Jones & Alcock, 2014;Jones & Inglis, 2015). However, an interesting case arose when CJ was applied to assess undergraduates' understanding of p values in the context of null-hypothesis significance testing (Bisson et al, 2016).…”

“…Where no standardised measure is available then another option, albeit one that adds time and expense to a study, is to develop a rubric for the open-ended test and compare the marks with CJ outcomes for a sample of student responses. The rubric should be developed iteratively with reference to a sample of student responses, and the marking conducted by two independent experts on a second sample of responses (see Jones & Inglis, 2015). However, this option can only provide a comparison of two methods of assessing the same responses and so is merely a proxy for criterion validity.…”

Who’s There? A Study of Students’ Reasoning about a Proof of Existence

“…Since it was not assumed in the study that either proof would convey upon the student absolute conviction -Ba stable psychological feeling of indubitability about that claim^(p. 16), the focus was necessarily on students' relative conviction -which, in the context of this work, was defined as that which (from the students' perspective) provides a greater degree of conviction. 7 Third, as noted in Jones and Inglis's (2015) work on comparative judgments:…”