Rater Sensitivity to Qualities of Lexis in Writing

Abstract

An examination of the scores given using analytic rating scales can provide insight into how well the scales are measuring what they are supposed to be measuring. Many analytic rating scales include separate scales for lexis and grammar, yet the distinction between lexis and grammar has not been widely researched or defined. It is therefore unclear whether separate lexis and grammar scales are appropriate. For this reason, the lexis scores for 140 timed essays on a Japanese university English proficiency test were analyzed to determine what portion of variance is accounted for by different lexical qualities. In this test, which was administered to incoming university students at the beginning of the academic year, it was found that lexical accuracy is predictive of lexis scores. The lexis scores, however, are predicted by the scores on the grammar scale much more than range, frequency, or even accuracy of lexis in the essays. The difficulties in separating lexis from grammar when rating writing are discussed. Finally, limitations of the study and suggestions for further research are given.

Language assessment and the inseparability of lexis and grammar: Focus on the construct of speaking

“…The data presented in this paper have additional implications for testing practice, as research suggests that raters frequently have difficulty in distinguishing vocabulary and grammar in the assessment of test taker performance. A study by Ruegg, Fritz, and Holland (2011) shows that raters of the Kanda English Proficiency Test are finding it difficult to keep lexis and grammar apart and often give the same score for both areas (see also Ruegg, 2015). Their findings suggest that an integrated approach that allows raters to give one score for lexicogrammar instead of two separate ones may facilitate the practice of scoring tests.…”

“…Their findings suggest that an integrated approach that allows raters to give one score for lexicogrammar instead of two separate ones may facilitate the practice of scoring tests. Although their focus is on assessing writing, Ruegg et al’s (2011) arguments may be transferable to the context of speaking assessment. A possible adjustment to rating scales would hence be to merge categories such as “Lexical Resource” and “Grammatical Resource” to create a “Lexicogrammatical Resource” category.…”

La competenza fraseologica in italiano lingua straniera tra i livelli elementare e intermedio

“…I due autori definiscono una tecnica, detta CollGram, nella quale ogni bigramma di un testo è associato alle due misure; attraverso tale tecnica gli autori intendono tracciare, tramite approcci longitudinali e pseudo-longitudinali, lo sviluppo della competenza fraseologica in apprendenti di lingua inglese. Il fondamento teorico di questa tecnica è quello di considerare grammatica e lessico come un continuum (v. tra gli altri Granger e Bestgen 2014; Römer 2009;Ruegg et al 2011). Poco, tuttavia, è detto circa i livelli inferiori e su altre lingue, come l'italiano (sul quale hanno lavorato, da prospettive diverse, Siyanova-Chanturia 2015; Siyanova-Chanturia e Spina 2015; Spina 2018).…”

An Experiment in the Ability of Raters to Evaluate Lexis in Writing

“…In a previous study, Ruegg, Fritz and Holland (2011) investigated raters' sensitivity to different lexical qualities when rating timed essays using the same analytic scale used in the present study. They found that although lexical accuracy did significantly predict lexis scores, grammar scores predicted lexis scores much more strongly than any other variable.…”

“…In a previous study (Ruegg, Fritz & Holland, 2011), it was observed that raters had difficulty separating lexis and grammar when using an analytic rating scale to assess timed essays. The current study further examines this issue by using an experimental design.…”