Developing the e-scape software system

Studies in Higher Education

Alcock

2013

111

115

Peer assessment typically requires students to judge peers' work against assessment criteria. We tested an alternative approach in which students judged pairs of scripts against one another in the absence of assessment criteria. First year mathematics undergraduates (N = 194) sat a written test of conceptual understanding of multivariable calculus, then assessed their peers' responses using pairwise comparative judgement. Inter-rater reliability was investigated by randomly assigning the students to two groups and correlating the two groups' assessments. Validity was investigated by correlating the peers' assessments with (i) expert assessments, (ii) novice assessments, and (iii) marks from other module tests. We found high validity and inter-rater reliability, suggesting that the students performed well as peer assessors. We interpret the results in the light of survey and interview feedback, and discuss directions for further research and development into the benefits and drawbacks of peer assessment without assessment criteria.

Section: Methodsmentioning

confidence: 99%

Peer assessment without assessment criteria

Studies in Higher Education

Alcock

2013

111

115

“…The implementation of CJ used for this study was TAG Development's e-scape system (Derrick, 2012), which presents pairs of scripts to experts online via an internet browser. The e-scape system in fact supports Adaptive Comparative Judgement (ACJ) using an algorithm similar to that used in traditional adaptive testing to reduce the required number of judgements (Pollitt, 2012).…”

Section: Methodsmentioning

confidence: 99%

Assessing Mathematical Problem Solving Using Comparative Judgement

Int J of Sci and Math Educ

Swan

Pollitt³

2014

There is an increasing demand from employers and universities for school leavers to be able to apply their mathematical knowledge to problem solving in varied and unfamiliar contexts. These aspects are however neglected in most examinations of mathematics and, consequentially, in classroom teaching. One barrier to the inclusion of mathematical problem solving in assessment is that the skills involved are difficult to define and assess objectively.We present two studies that test a method called comparative judgement (CJ) that might be well suited to assessing mathematical problem solving. CJ is an alternative to traditional scoring that is based on collective expert judgements of students' work rather than item-byitem scoring schemes. In Study 1 we used CJ to assess traditional mathematics tests and found it performed validly and reliably. In Study 2 we used CJ to assess mathematical problem-solving tasks and again found it performed validly and reliably. We discuss the implications of the results for further research and the implications of CJ for the design of mathematical problem-solving tasks.

“…The implementation of CJ used for the study was supported by TAG Development's e-scape system (Derrick, 2012), which presents pairs of students' work online via an internet browser and the examiner selects either the left or right student's work by clicking a button. The 750 pieces of student work were scanned and uploaded to the e-scape system.…”

Section: Comparative Judgementmentioning

confidence: 99%

The problem of assessing problem solving: can comparative judgement help?

Inglis

2015

Educ Stud Math

School mathematics examination papers are typically dominated by short, structured items that fail to assess sustained reasoning or problem solving. A contributory factor to this situation is the need for student work to be marked reliably by a large number of markers of varied experience and competence. We report a study that tested an alternative approach to assessment, called comparative judgement, which may represent a superior method for assessing open-ended questions that encourage a range of unpredictable responses. An innovative problem solving examination paper was specially designed by examiners, evaluated by mathematics teachers, and administered to 750 secondary school students of varied mathematical achievement. The students' work was then assessed by mathematics education experts using comparative judgement as well as a specially designed, resourceintensive marking procedure. We report two main findings from the research. First, the examination paper writers, when freed from the traditional constraint of producing a mark scheme, designed questions that were less structured and more problem-based than is typical in current school mathematics examination papers. Second, the comparative judgement approach to assessing the student work proved successful by our measures of inter-rater reliability and validity. These findings open new avenues for how school mathematics, and indeed other areas of the curriculum, might be assessed in the future.Keywords: assessment, problem solving, validity, comparative judgement 3 Typical mathematics exams are not fit for the purpose of assessing students' mathematical knowledge and skills. Analyses of the content and style of examination papers support the conjecture that mathematics examination papers comprise mainly short items that assess the rote learning of isolated facts and procedures (Berube, 2004; NCETM, 2009; Noyes, Wake, Drake & Murphy, 2011). An example question from a recent General Certificate for Secondary Education (GCSE) examination paper, a national qualification in England taken by most school leavers, illustrates the problem, as shown in Figure 1. ***FIGURE 1 ABOUT HERE*** At first glance, the question looks promising for assessing students' mathematical knowledge and skills. It makes use of a calendar context thereby appealing to the everyday relevance of mathematics. It also builds on students' experience of a counting system grouped in 7s to introduce an interesting generality that wherever the 2 by 2 square is positioned the provided algorithm will always give 7. However, to achieve full marks, all a student needs to do is compute the provided algorithm using the provided inputs. No explanation or proof of why the result is always 7 is required or rewarded. An efficient examination taker can achieve full marks without noticing there is a mathematically interesting generality at all.The question might be improved by asking students to compute the algorithm for a few 2 by 2 squares of their own choosing, and then asking them to explain what they ...