Variability in ESL Essay Rating Processes: The Role of the Rating Scale and Rater Experience

Barkaoui, Khaled

doi:10.1080/15434300903464418

Cited by 98 publications

(123 citation statements)

References 20 publications

Supporting

Mentioning

108

Contrasting

Unclassified

Order By: Relevance

“…On the one hand, organisation and argumentation were reflected upon most frequently. This complies with studies in absolute holistic scoring that also found that these components are prominent in the minds of assessors when assessing text quality (Barkaoui, 2010;Huot, 1993). However, in contrast to van Daal and colleagues (2016), the teachers in this assessment paid less attention to source use.…”

Section: Discussionsupporting

confidence: 49%

“…Moreover, those aspects valued mostly when comparing texts will be explored. Firstly, previous research has indicated that in the case of absolute holistic scoring, the content and organization of texts attracts most attention on the part of assessors (Barkaoui, 2010;Huot, 1993;Wolfe, 2006). Within the context of academic writing, van Daal and colleagues (2016) showed that, in particular, the problem statement and the extent that the sources were analysed and synthesized informed comparative judgements.…”

Section: Aims and Research Questionsmentioning

confidence: 99%

“…In addition, different scoring methods encourage assessors to look at other aspects of texts. For example, in the case of analytic scoring, the rating scales attracts particular attention during scoring, whereas in holistic scoring the text is more central (Barkaoui, 2010). Holistic scores are less task-specific, and therefore more generalizable than analytic scores (Bouwer et al, 2015;Schoonen, 2005;van den Bergh, De Maeyer, van Weijen, & Tillema, 2012).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

When teachers compare argumentative texts: Decisions informed by multiple complex aspects of text quality

Lesterhuis¹,

Daal²,

Gasse³

et al. 2018

L1 Educational Studies in Language and Literature

View full text Add to dashboard Cite

Insight into aspects that guide teachers' decisions when assessing student text quality is crucial to an understanding of the validity of text scores. Such research has been lacking in the context of comparative methods which are, however, increasingly being used for text assessment purposes. This study reports on the aspects of argumentative texts that guide Flemish teachers' decisions when using the comparative judgement method. In using this method, teachers indicate which text of a pair is of higher quality. 27 teachers explained 23 comparative judgements in a decision statement, when comparing randomly selected texts written by 135 students in their fifth year of general secondary education. This resulted in 596 statements referring to 2054 segments of aspects of text quality. Firstly, an inductive analysis revealed that teachers consider a wide range of aspects with regard to text quality when making comparison decisions. Secondly, the deductive aggregation of these aspects showed that most decisions are informed by the organisation and argumentation of the texts. Lastly, almost all statements reported complex aspects of text quality, whereas half of the decision statements also showed a reflection on the rule-applying aspects of text quality. We conclude that comparative judgement encourages teachers to make decisions on complex and multiple aspects of text quality. Further research should elaborate on whether the aspects that informed teachers' decisions are related to the text they choose, and whether teachers differ in the aspects to which they refer.

show abstract

Section: Discussionsupporting

confidence: 49%

Section: Aims and Research Questionsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

When teachers compare argumentative texts: Decisions informed by multiple complex aspects of text quality

Lesterhuis¹,

Daal²,

Gasse³

et al. 2018

L1 Educational Studies in Language and Literature

View full text Add to dashboard Cite

show abstract

“…On the other hand, the results of this study differ from those found by Barkaoui (2010), in which the use of rubrics was compared alongside teacher participants' level of experience, and the impact that these factors had on score variability was analyzed. It was found that the type of rubric used had a greater impact on rater reliability than years of teaching experience.…”

Section: Discussion Of Resultscontrasting

confidence: 54%

“…Bacha (2001), Barkaoui (2010) and Knoch (2009) have studied how distinct rubrics, mainly analytic vs. holistic rubrics, make a difference in raters' scoring. Khaled Barkaoui (2010) described how rater background and teaching experience, in comparison to the type of grade scale, influence grade variability.…”

Section: Introductionmentioning

confidence: 99%

Assessing EFL university students' writing: A study of score reliability

González

Trejo

Roux

2017

REDIE

View full text Add to dashboard Cite

The assessment of English as a Foreign Language (EFL) writing is a complex activity that is subject to human judgment, which makes it difficult to achieve a fair, accurate and reliable assessment of student writing (Pearson, 2004: 117;Hamp-Lyons, 2003). This study reports on the variability that exists between the analytical grades that 11 Mexican EFL university teachers awarded to five writing samples. It describes the raters' views on writing assessment and their use of analytical scoring rubrics. Data obtained from the grades awarded to each paper, and a background questionnaire, suggested that great variability was found between grades, and raters differed in their levels of leniency and severity, suggesting that having similar backgrounds and using the same rubric are not enough to ensure rater reliability. Participants' perceptions were found to be similar in terms of the use of analytical rubrics.Palabras: EFL writing, EFL writing assessment, Rater reliability, Analytical scoring rubric. ResumenLa evaluación de la escritura en inglés como lengua extranjera (EFL) es un proceso que depende del juicio humano, por lo que es difícil de obtener evaluaciones justas, acertadas y confiables (Pearson, 2004, p. 117;Hamp-Lyons, 2003). Este estudio reporta la variabilidad existente entre las calificaciones analíticas que 11 docentes Mexicanos universitarios de EFL proporcionaron a cinco trabajos escritos. Describe las percepciones de los participantes en torno a la evaluación de la escritura y el uso de las rúbricas de evaluación. Los datos obtenidos de las calificaciones de cada trabajo y de un cuestionario escrito revelaron que existe gran variedad entre las calificaciones proporcionadas y que los evaluadores difieren en sus niveles de exigencia, sugiriendo así que antecedentes homogéneos y el uso de una misma rúbrica 1 The authors presented preliminary results of this study at the 2014 TESL Canada Conference at the University of Regina, Saskatchewan, Canada, May 8-10, 2014 Assessing EFL university students' writing: a study of score reliability Palabras clave: Inglés como lengua extranjera, Evaluación de la escritura, Confiabilidad, Rúbrica de evaluación.

show abstract

Introspective Methods

Sasaki

2013

The Companion to Language Assessment

View full text Add to dashboard Cite

The chapter introduces three typical introspective measures (think‐aloud reports, immediate recalls, and stimulated recalls) that have been used in language assessment (LA) research mainly to elicit test takers' and test raters' thinking processes. First, it briefly touches upon how and why introspection became a legitimate research target in the humanities in general, then in applied linguistics, and later in the LA field. Because of the combined influences of behaviorism and structuralism after World War II, the LA field had long viewed the use of qualitative data such as introspection in research as somewhat suspicious. However, under the influence of information processing perspectives and the sociocultural orientation in applied linguistics, and most importantly through the redefinition of “test validity” in language testing itself, some recent LA studies have begun using introspection measures for the purpose of investigating how the test products (e.g., scores) are actually brought about. This evolution can also be related to the epistemological changes advocated by some researchers in the concept of assessment itself (e.g., dynamic assessment). The three measures listed above are operationally defined, and some possible advantages and disadvantages of each method when applied to LA research are discussed. Selected studies illustrate how these methods have been adopted in past research. The dos and don'ts that prospective researchers should bear in mind when planning to use these methods in their studies are also presented. The chapter concludes by mentioning several possible directions that future studies could take and that might benefit not only those involved in test development and administration but also those concerned with language education in general.

show abstract

Variability in ESL Essay Rating Processes: The Role of the Rating Scale and Rater Experience

Cited by 98 publications

References 20 publications

When teachers compare argumentative texts: Decisions informed by multiple complex aspects of text quality

When teachers compare argumentative texts: Decisions informed by multiple complex aspects of text quality

Assessing EFL university students' writing: A study of score reliability

Introspective Methods

Contact Info

Product

Resources

About