The reliability of setting grade boundaries using comparative judgement

Benton, Tom; Elliott, Gill

doi:10.1080/02671522.2015.1027723

Cited by 8 publications

(7 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other areas of focus have included peer assessment and feedback (Potter et al, 2017;Seery, Canty, & Phelan, 2012), the assessment of creative performances (Tarricone & Newhouse, 2016), and the assessment of oral narrative performances (Humphry, Heldsinger, & Dawkins, 2017). There has also been some research into the application of pairwise comparisons in setting grade boundaries (Benton & Elliot, 2016). To date, the internal consistency obtained using the method has typically ranged from high to very high, with most applications obtaining very high levels (Steedle & Ferrara, 2016).…”

Section: Introductionmentioning

confidence: 99%

A Two‐Stage Method for Classroom Assessments of Essay Writing

Humphry

Heldsinger

2019

J Educational Measurement

View full text Add to dashboard Cite

To capitalize on professional expertise in educational assessment, it is desirable to develop and test methods of rater‐mediated assessment that enable classroom teachers to make reliable and informative judgments. Accordingly, this article investigates the reliability of a two‐stage method used by classroom teachers to assess primary school students’ persuasive writing. Stage 1 involves pairwise comparisons and stage 2 involves rating against calibrated exemplars from stage 1 plus performance descriptors. A high level of interrater reliability among teachers was obtained. This is consistent with previous evidence that the two‐stage method is a viable classroom assessment method without extensive training and moderation. Implications for assessment practices in education are discussed with a focus on the widely expressed desire to value professional expertise.

show abstract

Section: Introductionmentioning

confidence: 99%

A Two‐Stage Method for Classroom Assessments of Essay Writing

Humphry

Heldsinger

2019

J Educational Measurement

View full text Add to dashboard Cite

show abstract

“…In this line, the scientific literature confirms the need to include rater effects in statistical modelling to ensure reliability and consistency of cut-off scores (Benton & Elliot, 2016. The use of predictions between cohorts of students examined in different years to ensure reliable standards in official examinations is most common in the UK (Benton & Sutch, 2014;Leckie & Baird, 2011).…”

Section: Antecedentsmentioning

confidence: 98%

Consistency analysis of assessment boards in University Entrance Examinations in Spain

Veas,

López-Pina

2024

Preprint

View full text Add to dashboard Cite

Introduction: University Entrance Examinations (EBAU in Spanish) is a nationwide program for assessing student achievement levels in Spain and determining access to public undergraduate degrees. Considering the need to measure the progress of rater performance, this paper analyses rating data from the June 2018 exam sitting in the Valencian Community, Spain. Methodology: A total of 54 assessment boards and 3000 students from five public universities were observed. The Many-Facet Rasch model (MFRM) was used as an extension of the one-parameter Rasch measurement model. All facets involved in analyses (examination board severity, subject difficulty, and group performance) were located on a common underlying linear scale. Results: Results showed large inconsistencies in the rating process with differences in severity levels of many subjects both within and between universities. Moreover, extreme differences were detected among subjects associated with health or biological sciences (Chemistry, Physics) as well as those related to the Humanities and Social Sciences (Latin or History of Spain). Conclusions: This study may serve as a starting point toward a reflective debate on the need to apply better criteria ensuring comparability of examination standards in Spain.

show abstract

“…The starting place for the revival of Comparative Judgement in educational research was in the context of the England's upper secondary school examination system and specifically in terms of maintaining high-stakes examination standards over time, e.g., ensuring that a 'C grade' on the A-level mathematics examination in 2022 represents the same standard of achievement as the equivalent grade in previous years, as such standard maintaining is known to be affected by subjectivity and unreliability in expert judgements (Benton, 2021;Benton & Elliot, 2016;Black & Bramley, 2008;Bramley, 2005Bramley, , 2007Bramley, Bell & Pollitt, 1998;Bramley & Gill, 2010;Leech, Gill, Hughes, & Benton, 2022;Pollitt & Murray, 1996;Seery, Kimbell, & Buckley, 2022). Moreover, England's examination system is unique by international standards in that these high-stakes examinations are created by several different organisations, known as examination boards, which further complicates the process of ensuring comparable examination standards, as they must also be maintained across these boards.…”

Section: Educational Applications Of Comparative Judgementmentioning

confidence: 99%

“…Moreover, the relative nature of the judgements has been argued to be cognitively simpler for assessors than making categorisations when making judgements about complex constructs, which in turn enhances the internal consistency of the judgements (Pollitt, 2012). Benton and Gallagher (2018) put forward an alternative view that Comparative Judgement's high level of reliability is simply a by-product of its requirement for multiple judgements on each artefact, whereas other assessment approaches such as providing a rating typically only involve a single or small number of judgements for each artefact. This conjecture is consistent with other empirical findings that have demonstrated a high association between the reliability of the Comparative Judgement scale scores and the number of comparisons per artefact (Crompvoets, Béguin, & Sijtsma, 2020;Goossens & De Maeyer, 2017;Verhavert, Bouwer, Donche, & De Maeyer, 2019).…”

Section: Reliability and Validity Of Comparative Judgementmentioning

confidence: 99%

Comparative Judgement

McGrane

2024

Preprint

View full text Add to dashboard Cite

In the early 20th Century, Thurstone introduced the Law of Comparative Judgment as a scaling model for the method of paired comparisons, but it is only in the past 30 years that it has re-emerged as a mainstream approach in educational assessment. This re-emergence has particularly taken place in contexts where complex constructs are assessed and where expert judgment and holistic evaluation are valued, including examination standard maintenance, and the assessment of writing, design, and the creative arts. Technological advancements have furthered this revival by streamlining the otherwise laborious method through the distribution of subsets of pairs across judges and using selective algorithms to prioritize the most informative pairwise judgments. An extensive review of the literature has shown Comparative Judgment to be a highly reliable assessment method for a wide range of constructs. Moreover, it has been shown in certain contexts to have good validity evidence in terms of concurrent and criterion validity, as well as being highly sensitive to construct-relevant variance, although further research is required to validate the method across a broader range of constructs. Given the proliferation of research into Comparative Judgment in just the past decade, the next 30 years should prove to be even more fruitful in terms of the scope of its educational applications and further enhancement of its reliability and validity across these applications.

show abstract

The reliability of setting grade boundaries using comparative judgement

Cited by 8 publications

References 15 publications

A Two‐Stage Method for Classroom Assessments of Essay Writing

A Two‐Stage Method for Classroom Assessments of Essay Writing

Consistency analysis of assessment boards in University Entrance Examinations in Spain

Comparative Judgement

Contact Info

Product

Resources

About