Effects of marking method and rater experience on ESL essay scores and rater performance

Barkaoui, Khaled

doi:10.1080/0969594x.2010.526585

Cited by 61 publications

(99 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Among studies concerning rater background, the issue of native and non-native speakers (NNS) and their rating patterns have been explored by multiple scholars. Some studies found significant differences based on the L1 background (Zhang and Elder 2014;Gui, 2012) while others found no significant differences (Johnson and Lim's 2009;Barkaoui, 2011). Research that found a minimal effect of L1 language background had NNS raters who possessed near-native language fluency (Johnson and Lim, 2009;Barkaoui, 2011).…”

Section: Different Rater Background Effectsmentioning

confidence: 99%

“…Some studies found significant differences based on the L1 background (Zhang and Elder 2014;Gui, 2012) while others found no significant differences (Johnson and Lim's 2009;Barkaoui, 2011). Research that found a minimal effect of L1 language background had NNS raters who possessed near-native language fluency (Johnson and Lim, 2009;Barkaoui, 2011). Johnson and Lim's (2009) study looked at the rater language background effect on the rating of performance assessment in the MELAB writing test.…”

Section: Different Rater Background Effectsmentioning

confidence: 99%

“…Performance-based assessment, when done with human raters, is a time-consuming and expensive task; therefore, administrators want to find a scale that can reduce rating time and cost while ensuring valid and reliable results. While there has been research on the effects of tasks and raters on ratings (Schoonen, 2005), there has been limited research (Bacha, 2001;Barkaoui, 2007Barkaoui, , 2010Barkaoui, , 2011O'Loughlin, 1994;Song & Caruso, 1996) on how different scales impact rating performance and scores. For this reason, I compare two different scale types (binary and analytic) that were developed on the same assessment construct (i.e., paragraph structure, content, form, and vocabulary).…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Writing scale effects on raters: an exploratory study

Jeong¹

2019

Lang Test Asia

View full text Add to dashboard Cite

In writing assessment, finding a valid, reliable, and efficient scale is critical. Appropriate scales, increase rater reliability, and can also save time and money. This exploratory study compared the effects of a binary scale and an analytic scale across teacher raters and expert raters. The purpose of the study is to find out how different scale types impact rating performance and scores. The raters in this study rated twenty short EFL essays using the two scales, completed a rater cognition questionnaire, and took part in an in-depth interview. The ratings were analyzed using a multi-faceted Rasch analysis to compare essay scores and rater statistics across scales and rater groups. The results indicated when using the binary scale, the raters spent less time and were less spread out and more consistent in their ratings. Three out of four raters replied that less mental effort was required when using the binary scale and felt more confident in their ratings. Across the two rater groups, there was a bigger shift in rating performance when using the binary scale for the teacher raters than the expert raters. This implies that scale design had a greater effect on teacher raters. The overall findings suggest that the binary scale maybe a better fit for large scale assessment with sufficient rater training.

show abstract

Section: Different Rater Background Effectsmentioning

confidence: 99%

Section: Different Rater Background Effectsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Writing scale effects on raters: an exploratory study

Jeong¹

2019

Lang Test Asia

View full text Add to dashboard Cite

show abstract

“…Studies focusing on the causes of inconsistency amongst assessors are increasingly gaining popularity in general educational research (e.g., Hunter & Docherty, 2011;Price et al, 2011;Shay, 2008) and in language education (see Barkaoui, 2011). These researchers have to date highlighted different factors leading to inconsistency in awarding assessment scores such as ambiguous modes of assessment or assessment criteria (O'Donovan, Price & Rust, 2004), the absence of disciplinary knowledge as anchorage point for our assessment processes (Shay, 2008;Price, 2005), the effect of tacit beliefs (Hunter & Docherty, 2011) and marking method and rater experience (Barkaoui, 2011).…”

mentioning

confidence: 99%

Community of assessment practice or interests: The case of EAP writing assessment

Al-Maamari

2016

Indonesian J. Appl. Linguist

View full text Add to dashboard Cite

Setting, disseminating and applying assessment standards are part of university academic programmes of study. Nowadays, assessment is increasingly viewed from a social practice perspective, and so doing entails exploring how the quality of assessment is shaped by interaction and co-participation with different communities of practice. Therefore, based on this perspective, the study reported here aimed to examine the assessment policies and practices of laboratory report writing of first year students in credit-bearing, English for Special Purposes programmes at a university in the Sultanate of Oman. Interviews of programme administrators and the instructors plus institutional and programme documents were examined to investigate these assessment policies and practices. The programme administrators were asked about how they planned the written assessment in their programmes, and the instructors were asked about their experiences of these assessments. The data were then analysed thematically using community of practice framework, namely in relation to (1) a shared repertoire of communal resources, (2) mutual engagement, and (3) a sense of joint enterprise. It was found that instead of community of practice, there were (sub)communities of practices wherein interaction, negotiation and communication amongst members and non-members were punctuated by control, power and autonomy, all working with the aim of narrowing the range between the personal goals of the academic and the communal goals of the institution. The overarching conclusion is that in their assessment practices, the two instructional programmes exhibited varying degrees of community of practice based on the above three attributes.

show abstract

“…Although the influence of rating procedures on the scores assigned by raters is acknowledged (e.g. , little research has been done on the effects of different marking methods (Barkaoui, 2011) and the results include both arguments for and against different rating methods (Cushing Weigle, 2002;Rijlaarsdam et al, 2012). This study aims at improving the reliability and validity of a large scale writing assessment in primary education by means of incorporating anchor essays into an analytical rating procedure.…”

Section: Outlinementioning

confidence: 99%

Assessing writing ability in primary education : on the evaluation of text quality and text complexity

Feenstra¹

View full text Add to dashboard Cite

Effects of marking method and rater experience on ESL essay scores and rater performance

Cited by 61 publications

References 21 publications

Writing scale effects on raters: an exploratory study

Writing scale effects on raters: an exploratory study

Community of assessment practice or interests: The case of EAP writing assessment

Assessing writing ability in primary education : on the evaluation of text quality and text complexity

Contact Info

Product

Resources

About