Evaluating Comparative Judgment as an Approach to Essay Scoring

Steedle, Jeffrey T.; Ferrara, Steve

doi:10.1080/08957347.2016.1171769

Cited by 47 publications

(49 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One of these methods is comparative judgement, wherein teachers have to compare texts with one another. These comparative methods seem to be easier and more promising in terms of obtaining reliable scores (Bramley, 2007;Steedle & Ferrara, 2016). However, studies elaborating on the aspects that the teachers take into account when comparing texts are scarce.…”

Section: Aims and Research Questionsmentioning

confidence: 99%

“…The quality of each text is then calculated by analyzing all the comparative judgements made by all assessors (more information on the method can be found in Lesterhuis, Verhavert, Coertjens, Donche & De Maeyer, 2016, and Pollitt, 2012a, 2012b. Given that comparing texts is an easier and more reliable task when it comes to assessing complex issues such as text quality (Gill & Bramley, 2013), comparative methods have been increasingly appraised (Bramley, 2007;Heldsinger & Humphry, 2010;Pollitt, 2012a;Steedle & Ferrara, 2016). However, little is known about which aspects of text quality assessors consider when comparing texts, which hampers a valid interpretation of the assessment outcomes.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

When teachers compare argumentative texts: Decisions informed by multiple complex aspects of text quality

Lesterhuis¹,

Daal²,

Gasse³

et al. 2018

L1 Educational Studies in Language and Literature

View full text Add to dashboard Cite

Insight into aspects that guide teachers' decisions when assessing student text quality is crucial to an understanding of the validity of text scores. Such research has been lacking in the context of comparative methods which are, however, increasingly being used for text assessment purposes. This study reports on the aspects of argumentative texts that guide Flemish teachers' decisions when using the comparative judgement method. In using this method, teachers indicate which text of a pair is of higher quality. 27 teachers explained 23 comparative judgements in a decision statement, when comparing randomly selected texts written by 135 students in their fifth year of general secondary education. This resulted in 596 statements referring to 2054 segments of aspects of text quality. Firstly, an inductive analysis revealed that teachers consider a wide range of aspects with regard to text quality when making comparison decisions. Secondly, the deductive aggregation of these aspects showed that most decisions are informed by the organisation and argumentation of the texts. Lastly, almost all statements reported complex aspects of text quality, whereas half of the decision statements also showed a reflection on the rule-applying aspects of text quality. We conclude that comparative judgement encourages teachers to make decisions on complex and multiple aspects of text quality. Further research should elaborate on whether the aspects that informed teachers' decisions are related to the text they choose, and whether teachers differ in the aspects to which they refer.

show abstract

Section: Aims and Research Questionsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

When teachers compare argumentative texts: Decisions informed by multiple complex aspects of text quality

Lesterhuis¹,

Daal²,

Gasse³

et al. 2018

L1 Educational Studies in Language and Literature

View full text Add to dashboard Cite

show abstract

“…Pollitt (2012) suggested 12 rounds as a target for achieving a reliable rank order while Bartholomew, Strimel, & Zhang's (2017) suggested that a reliable rank order may be achieved as early as 6 rounds if ACJ were utilized by a solitary assessor (P). Steedle and Ferrara (2016) reported that reliability was above 0.8 by 9 rounds of judgment and slowly increased with further judgments (L).…”

Section: Acj Implementation Resultsmentioning

confidence: 99%

“…Speaking about efficiency, and the ACJ algorithm that guides the process, Steedle and Ferrara (2016) laid out a synthesis of studies that involved comparative judgments of student performance and concluded that the assessment process could become more efficient using adaptive comparative judgment and the associated algorithm for determining pairs (L). This argument for ACJ over CJ, in terms of efficiency, resides in the pairs for comparative judgment being paired more optimally and efficiently-instead of having random comparisons, there are controlled, intentional pairings that makes the process more efficient in obtaining high levels of reliability (C. Rangel & M. Wingfield, personal communication, August 29, 2017).…”

Section: Acj Implementation Resultsmentioning

confidence: 99%

See 1 more Smart Citation

Untitled

2018

RMS

View full text Add to dashboard Cite

Contributors PrefaceWelcome to the inaugural issue of the Research Monograph Series (RMS) -an annual publication dedicated to supporting new and emerging scholars in establishing their presence within the realm of academia through contributions to global discussions on science, technology, engineering, and mathematics (STEM) education reform. As a refereed publication sponsored by the Research and Scholarship (R&S) Committee of the Council on Technology and Engineering Teacher Education (CTETE), the new RMS aims to promote serious scholarly discussion of the connections among the Technology Education (TE), Technology and Design Education (TDE), and Technology and Engineering Education (T&EE) fields, and their place within the global discourse regarding the teaching and learning issues accompanying the steady increased focus on the integration of subjects through STEM education.The intent for technological studies to serve as an integrator of content and practice (knowledge) from many subjects is at the core of the Standards for Technological Literacy first published in 2000 (ITEA, pp. 6-9). This pedagogical intent was made clear in the statement that "…although the study of technology may sometimes be a separate subject, it can never be an isolated subject, cut off from the rest of the curriculum" (p. 9), and which echoes that same intent by the AAAS who nearly a decade earlier recognized that "…the ideas and practice of science, mathematics, and technology are so closely intertwined that we do not see how education in any one of them can be undertaken in isolation from the others" (1993, pp. 321-322). In the nearly 20 years since the STL were published, the field has developed and implemented a plethora of programs and curricula designed specifically to integrate and intentionally teach the content and practices of the many subjects inherent within technological and/or engineering design. Through these efforts the profession developed a significant body of unique knowledge and expertise regarding sound integrative practices.Throughout the past decade there has been a concerted effort at both elementary and secondary levels across all subjects, but particularly science and mathematics, to increasingly incorporate the use of technological and/or engineering design based learning (T/E DBL) as part of their instructional approach. Few would argue against the potential benefits for teaching and learning within such efforts. However, as potentially beneficial as they may be, there are many issues surrounding such incorporation -validity of T/E DBL as an efficacious strategy, pedagogical fidelity in how other subjects teach technological/engineering design, adequate T/E content knowledge, adequate T/E pedagogical content knowledge, etc. -and around which educational researchers all over the world and in all STEM fields are beginning to address.The Research Monograph Series is an endeavor to provide a rigorous, peer reviewed publication platform for emerging STEM scholars around the world to address these issues by voi...

show abstract

STEM Skill Assessment: An Application of Adaptive Comparative Judgment

Bartholomew

Williams

2020

Advances in STEM Education

View full text Add to dashboard Cite

Evaluating Comparative Judgment as an Approach to Essay Scoring

Cited by 47 publications

References 14 publications

When teachers compare argumentative texts: Decisions informed by multiple complex aspects of text quality

When teachers compare argumentative texts: Decisions informed by multiple complex aspects of text quality

Untitled

STEM Skill Assessment: An Application of Adaptive Comparative Judgment

Contact Info

Product

Resources

About