Exploring Variability in Judging Writing Ability in a Second Language: A Study of Four Experienced Raters of Esl Compositions

Erdosy, Muhammad Usman

doi:10.1002/j.2333-8504.2003.tb01909.x

Cited by 28 publications

(35 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…An overview of the relevant assessment literature shows that a variety of qualitative and quantitative tools like introspective and retrospective think aloud protocols (e.g., Cumming et al 2001Cumming et al , 2002Erdosy 2004), group or individual interviews (e.g., Erdosy 2004), written score explanations (e.g., Barkaoui 2010;Milanovic et al 1996;Rinnert and Kobayashi 2001;Siddiqui 2016), questionnaires (e.g., Shi 2001) and panel discussions (e.g., Kuiken and Vedder 2014) have been used by different researchers to find an answer to the Research Questions 1 and 2 outlined in the next section. Think Aloud Protocols (TAPs) or verbal protocols, for instance, which require the participants to verbalise their thoughts while they are actually rating, are the most widely used methods to investigate the rating process of essays in English as a first language (Huot 1993;Wolfe et al 1998) as well as in English as a second language context (Cumming et al 2001;Lumley 2005).…”

Section: Literature Reviewmentioning

confidence: 99%

Markers’ criteria in assessing English essays: an exploratory study of the higher secondary school certificate (HSCC) in the Punjab province of Pakistan

Fernández

Siddiqui

2017

Lang Test Asia

View full text Add to dashboard Cite

Background: Marking of essays is mainly carried out by human raters who bring in their own subjective and idiosyncratic evaluation criteria, which sometimes lead to discrepancy. This discrepancy may in turn raise issues like reliability and fairness. The current research attempts to explore the evaluation criteria of markers on a national level high stakes examination conducted at 12th grade by three examination boards in the South of Pakistan. Methods: Fifteen markers and 30 students participated in the study. For this research, data came from quantitative as well as qualitative sources. Qualitative data came in the form of scores on a set of three essays that all the fifteen markers in the study marked. For the purpose of this study, they weren't provided with any rating scale as to replicate the current practices. Qualitative data came from semi-structured interviews with the selected markers and short written commentaries by the markers to rationalize their scores on the essays. Results: Many-facet Rasch model analyses present differences in raters' consistency of scoring and the severity they exercised. Additionally, an analysis of the interviews and the commentaries written by raters justifying the scores they gave showed that there is a great deal of variability in their assessment criteria in terms of grammar, attitude towards mistakes, handwriting, length, creativity and organization and use of cohesive devices. Conclusions: The study shows a great deal of variability amongst markers, in their actual scores as well as in the criteria they use to assess English essays. Even they apply the same evaluation criteria, markers differ in the relative weight they give them.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Markers’ criteria in assessing English essays: an exploratory study of the higher secondary school certificate (HSCC) in the Punjab province of Pakistan

Fernández

Siddiqui

2017

Lang Test Asia

View full text Add to dashboard Cite

show abstract

“…As Erdosy (2004) states, "constructs such as 'content' and 'organization' have as many manifestations as there are raters" (p. 10). Furthermore, even if the constructs are defined in rubrics, each rater interprets the constructs differently (MacIntyre, 2007).…”

Section: Literature Review Defining Organizationmentioning

confidence: 99%

“…A similar study conducted by Erdosy (2004) examined think-aloud protocols of four experienced raters of the TOEFL TWE and compared the ways they evaluated a written text and how their backgrounds affected their ratings. In the protocols, teachers identified cohesive devices, length, and rigid paragraph structure to be important when judging writing proficiency.…”

Section: Literature Review Defining Organizationmentioning

confidence: 99%

See 1 more Smart Citation

Organization of ideas in writing: what are raters sensitive to?

Ruegg

Sugiyama

2013

Language Testing in Asia

View full text Add to dashboard Cite

Whether foreign language writing is rated using analytic rating scales or holistically, the organization of ideas is invariably one of the aspects assessed. However, it is unclear what raters are sensitive to when rating writing for organization. Which do they value more highly, the physical aspects of organization, such as paragraphing and the existence of organization markers, or deeper textual aspects, such as the coherent flow of ideas? This study investigates whether raters of timed essays value paragraphing, cohesive devices or coherence more when assigning a score for organization. The current study used multiple regression to ascertain what raters of the writing section of an in-house proficiency test were sensitive to when rating writing for organization using an analytic rating scale. The number of paragraphs, number of cohesive devices and coherence within 116 timed essays were evaluated and it was found that raters seem to value the physical aspects of organization more than the deeper textual aspects. Specifically, the number of paragraphs and the number of cohesive devices predicted differences in scores assigned for organization. In addition, the scores assigned for content were significantly predictive of scores for organization.

show abstract

“…Raters are found to vary in assessing linguistic features (Bejar, 2006;Elder, 1993) and frequently deviate from rating criteria (Eckes, 2005;Erdosy, 2003;Taylor & Jones, 2001). May (2006), for instance, found more than 30% of raters' comments subsumed additional features that were not mentioned in the criteria.…”

Section: Introductionmentioning

confidence: 99%

How Do Raters Judge Spoken Vocabulary?

Li¹

2016

ELT

View full text Add to dashboard Cite

The aim of the study was to investigate how raters come to their decisions when judging spoken vocabulary. Segmental rating was introduced to quantify raters' decision-making process. It is hoped that this simulated study brings fresh insight to future methodological considerations with spoken data. Twenty trainee raters assessed five Chinese students' monologic texts on vocabulary in this study. Both segmental rating and overall rating were retrieved from the raters. Rasch analysis suggested variation between raters in their judgment of vocabulary, although consistency was found in general. Besides, there was a mismatch between candidates' vocabulary scores and their lexical statistics. The raters' decision-making process was generally cumulative.

show abstract

Exploring Variability in Judging Writing Ability in a Second Language: A Study of Four Experienced Raters of Esl Compositions

Abstract: Variability in judgments of ESL compositions is inherent in the view that raters are "readers" with prior experiences. Such a view, however, obliges researchers to understand how personal background and professional experience influence both scoring procedures and scoring criteria.

Cited by 28 publications

References 33 publications

Markers’ criteria in assessing English essays: an exploratory study of the higher secondary school certificate (HSCC) in the Punjab province of Pakistan

Markers’ criteria in assessing English essays: an exploratory study of the higher secondary school certificate (HSCC) in the Punjab province of Pakistan

Organization of ideas in writing: what are raters sensitive to?

How Do Raters Judge Spoken Vocabulary?

Contact Info

Product

Resources

About