Gains in marking reliability from item-level marking: is the sum of the parts better than the whole?

Wheadon, Christopher; Moira, Anne Pinot de

doi:10.1080/13803611.2013.828629

Cited by 5 publications

(4 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Adaptive Comparative Judgement was administered via the web with the free tool, "Mo More Marking" [8]. The workflow process went as follows: students submitted a group paper via the web with no names or identifying features included.…”

Section: Methodsmentioning

confidence: 99%

Using Adaptive Comparative Judgement to Assess Student Work in an MBA Course

Metzgar¹

2016

IJI

View full text Add to dashboard Cite

show abstract

Section: Methodsmentioning

confidence: 99%

Using Adaptive Comparative Judgement to Assess Student Work in an MBA Course

Metzgar¹

2016

IJI

View full text Add to dashboard Cite

show abstract

“…Research on the marking of writing of older students has found similarly poor marking consistency, with levels comparable in England to internationally (Rhead et al, 2018). While there is almost no evidence regarding the consistency of 3 direct classification of writing into grades rather than a fine mark, there is no theoretical reason to believe that a broad classification would be more reliable than a fine mark that is then converted into a grade (Wheadon & Pinot de Moira, 2013). Attempts to improve mark schemes and rubrics have led to no reported gains in marking consistency, rather issues arise with hyper-specificity distorting approaches to teaching and learning (Cuff et al, 2018;Humphry & Heldsinger, 2014;Rezaei & Lovorn, 2010).…”

Section: Introductionmentioning

confidence: 99%

The classification accuracy and consistency of comparative judgement of writing compared to rubric-based teacher assessment

Moira¹,

Wheadon²,

Christodoulou³

2022

Research in Education

Self Cite

View full text Add to dashboard Cite

Writing is generally assessed internationally using rubric-based approaches, but there is a growing body of evidence to suggest that the reliability of such approaches is poor. In contrast, comparative judgement studies suggest that it is possible to assess open ended tasks such as writing with greater reliability. Many previous studies, however, have failed to provide direct comparisons between these approaches as the reliability measures for rubric- and marking-based studies are not comparable with the internal measures of reliability cited by comparative judgement studies. We investigated the classification accuracy and consistency of a rubric-based approach to the grading of writing with a comparative judgement approach. The writing was gathered from 11-year-olds in low stakes settings in England. We present evidence that the comparative judgement approach has twice the classification accuracy of the rubric-based approach and is perfectly viable in terms of its efficiency. We discuss the limitations of the comparisons and consider what a national system for assessing writing based on a comparative judgement approach could look like.

show abstract

“…Research on the marking of writing of older students has found similarly poor marking consistency, with levels comparable in England to internationally (Rhead et al, 2018). While there is almost no evidence regarding the consistency of direct classification of writing into grades rather than a fine mark, there is no theoretical reason to believe that a broad classification would be more reliable than a fine mark that is then converted into a grade (Wheadon & Pinot de Moira, 2013). Attempts to improve mark schemes and rubrics have led to no reported gains in marking consistency, rather issues arise with hyper-specificity distorting approaches to teaching and learning (Cuff et al, 2018;Humphry & Heldsinger, 2014;Rezaei & Lovorn, 2010).…”

Section: Introductionmentioning

confidence: 99%

The classification accuracy and consistency of comparative judgement of writing compared to rubric-based teacher assessment

Wheadon¹,

Moira²,

Christodoulou³

2020

Preprint

Self Cite

View full text Add to dashboard Cite

Writing is generally assessed internationally using rubric-based approaches, but there is a growing body of evidence to suggest that the reliability of such approaches is poor. In contrast, comparative judgement studies suggest that it is possible to assess open ended tasks such as writing with greater reliability. Many previous studies, however, have failed to provide direct comparisons between these approaches as the reliability measures for rubric- and marking-based studies are not comparable with the internal measures of reliability cited by comparative judgement studies. We investigated the classification accuracy and consistency of a rubric-based approach to the grading of writing with a comparative judgement approach. The writing was gathered from 11-year-olds in low stakes settings in England. We present evidence that the comparative judgement approach has twice the classification accuracy as the rubric-based approach and is perfectly viable in terms of its efficiency. We discuss the limitations of the comparisons and consider what a national system for assessing writing based on a comparative judgement approach could look like.

show abstract

Gains in marking reliability from item-level marking: is the sum of the parts better than the whole?

Cited by 5 publications

References 10 publications

Using Adaptive Comparative Judgement to Assess Student Work in an MBA Course

Using Adaptive Comparative Judgement to Assess Student Work in an MBA Course

The classification accuracy and consistency of comparative judgement of writing compared to rubric-based teacher assessment

The classification accuracy and consistency of comparative judgement of writing compared to rubric-based teacher assessment

Contact Info

Product

Resources

About