Effect of genre on the generalizability of writing scores

Abstract

In the present study, aspects of the measurement of writing are disentangled in order to investigate the validity of inferences made on the basis of writing performance and to describe implications for the assessment of writing. To include genre as a facet in the measurement, we obtained writing scores of 12 texts in four different genres for each participating student. Results indicate that across raters, tasks and genres, only 10% of the variance in writing scores is related to individual writing skill. In order to draw conclusions about writing proficiency, students should therefore write at least three different texts in each of four genres rated by at least two raters. Moreover, when writing scores are obtained through highly similar tasks, generalization across genres is not warranted. Inferences based on text quality scores should, in this case, be limited to genre-specific writing. These findings replicate the large task variance in writing assessment as consistently found in earlier research and emphasize the effect of genre on the generalizability of writing scores. This research has important implications for writing research and writing education, in which writing proficiency is quite often assessed by only one task rated by one rater.

The Effect of Writing Task and Task Conditions on Colombian EFL Learners’ Language Use

“…The findings provide additional support for claims that writing task type plays an important role when assessing the quality and generalizability of written texts (e.g., Bouwer, Béguin, Sanders, & van den Bergh, 2015), and sheds further light on potential differences in L2 writing associated with paragraph and essay types (e.g., McDonough et al, in press;Shi, 1998). In terms of task conditions, whether students wrote individually or collaboratively did not affect their paragraph ratings or clause use for either task type.…”

The Advantages and Disadvantages of Using Qualitative and Quantitative Approaches and Methods in Language “Testing and Assessment” Research: A Literature Review

“…It also fails to ascertain deeper underlying meanings and explanations. The study of Bouwer, Béguin, Sanders and van den Bergh (2015) measured the effect of genre on writing score involving various types of variance such as person, genre, person by genre, task within genre, person by task within genre, raters who rated tasks within different genres, and random error. But the study has not explained the reasons of the effect and their meanings in that particular context.…”

When teachers compare argumentative texts: Decisions informed by multiple complex aspects of text quality

“…These methods support assessors' holistic scoring by providing them with texts to compare. These texts can be exemplar texts representing certain levels of quality (Bouwer, Béguin, Sanders, & van den Bergh, 2015), or other texts that also need to be scored. The latter leads to relative measures of text quality, as is the case with comparative judgement.…”

“…For example, in the case of analytic scoring, the rating scales attracts particular attention during scoring, whereas in holistic scoring the text is more central (Barkaoui, 2010). Holistic scores are less task-specific, and therefore more generalizable than analytic scores (Bouwer et al, 2015;Schoonen, 2005;van den Bergh, De Maeyer, van Weijen, & Tillema, 2012). To date, not much is known about which aspects guide assessors' decisions when using comparative methods.…”