Using the method of pairwise comparison to obtain reliable teacher assessments

Abstract

Demands for accountability have seen the implementation of large scale testing programs in Australia and internationally. There is, however, a growing body of evidence to show that externally imposed testing programs do not have a sustained impact on student achievement. It has been argued that teacher assessment is more effective in raising student achievement levels. However, it is also often argued that teacher assessments are less reliable than the results of testing programs. This paper presents a study in which teachers judged writing scripts using the process of pairwise comparison to generate a scale. The analysis showed high internal consistency of the teacher judgements. The scale locations from pairwise comparisons were highly correlated with scale estimates for the same students from a large-scale testing program. The results demonstrate it is possible to efficiently obtain highly reliable and valid teacher judgements using the process of pairwise comparison. Reliability indices are also provided for a series of small-scale assessments that used the same methodology in a range of other domains. The results support the findings of the main study. The article discusses the benefits of using the method to supplement and validate results from large-scale testing programs.

The problem of assessing problem solving: can comparative judgement help?

“…CJ has been used in a range of educational research and practice (e.g., Bramley, 2007;Bramley, Bell & Pollitt, 1998;Heldsinger & Humphry, 2010;Seery, Canty & Phelan, 2012). …”

“…Indeed the development of CJ for educational assessment has involved a diversity of disciplines ranging from design and technology (Kimbell, 2012) to narrative writing (Heldsinger & Humphry, 2010). Therefore, CJ may offer the potential to enable the assessment of rich and authentic educational outcomes in a wide variety of subject areas and contexts.…”

“…The experts' decisions are fitted to a statistical model to produce a standardised parameter estimate (z-score) for each student (Pollitt, 2012a). The parameter estimates are then used to construct a scaled rank order of student work from "worst" to "best" and the usual assessment arrangements, such as allocating grades, can be applied to the rank order (see Jones & Alcock, 2014;McMahon & Jones, 2014).CJ has been used in a range of educational research and practice (e.g., Bramley, 2007;Bramley, Bell & Pollitt, 1998;Heldsinger & Humphry, 2010; Seery, Canty & Phelan, 2012).Thurstone's (1927) underlying principle of comparative judgement suggests that we should obtain reliable assessment outcomes even though the process is based on "subjective"judgements. In previous studies we have found this to be the case when used to assess One potential contribution that CJ might offer education is its suitability for assessing nebulous constructs that are deemed important but which are difficult to specify comprehensively in mark schemes (Pollitt, 2012b).…”

Using comparative judgement and online technologies in the assessment and measurement of creative performance and capability

“…Pollitt posited that in Kimbell et al's (2009) study the high reliability coefficient of 0.96 generated by 28 judges assessing 352 e-portfolios with 3067 judgements was higher than any analytical marking system could achieve. Studies by Heldsinger and Humphry (2010), (2013) found that highly reliable comparative judgements of writing scripts by a large number of judges were made using calibrated exemplars as referents, as suggested by Thurstone (1928). They argued that comparative judgement, using calibrated exemplars, were successfully used as a highly reliable method to validate results from large-scale testing programs without the extensive assessor training and moderation processes that were required for absolute analytical judgements.…”

“…Several researchers have found that judgements based on quality, using specified holistic criteria, resulted in a more valid assessment of performance (Bramley, 2007;Heldsinger & Humphry, 2010;Pollitt, 2012a). According to Pollitt (2012a) the high reliability of comparative judgement is a "consequence of the constant focus on validity" (p. 168) and this "demands and checks that a sufficient consensus exists amongst the pool of judges involved" (p. 167).…”