The association between SAT prompt characteristics, response features, and essay scores

Kobrin, Jennifer L.; Deng, Hui; Shaw, Emily J.

doi:10.1016/j.asw.2011.01.001

Cited by 38 publications

(17 citation statements)

References 23 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…This way only three essays from the 179 were discarded (one from the B1 and two from the C1 bands). We used McCarthy's gramulator to select 200 words from the middle of each essay to maximize opportunities for inclusion of parts of the beginning, the middle and end of the essays in the chosen extract, whilst avoiding sentences with copies of the prompt, which is a strategy in particular of lower level learners (Kobrin, Deng, and Shaw 2011). Such copies of lexical and syntactic structures from the prompt were found to occur most frequently at the start of essays.…”

Section: Methodsmentioning

confidence: 99%

Back to Basics: How Measures of Lexical Diversity Can Help Discriminate between CEFR Levels

Treffers‐Daller¹,

Parslow

Williams

2016

Applied Linguistics

View full text Add to dashboard Cite

This study contributes to ongoing discussions on how measures of lexical diversity (LD) can help discriminate between essays from second language learners of English, whose work has been assessed as belonging to levels B1 to C2 of the Common European Framework of 1 We are very grateful to Pearson Education Ltd for sponsoring us with a research grant that made this study possible, and to Kirsten Ackermann, Veronica Benigno and Jeremy Hancock for their support in carrying out this project. 2 This work was funded by Language Testing, a trading division of Pearson Education Ltd.We are very grateful to Kirsten Ackermann, Veronica Benigno and Jeremy Hancock for their support in carrying out this project. We are also indebted to Scott Jarvis for providing a spreadsheet to help us with computing HD-D. Reference (CEFR). The focus is in particular on how different operationalisations of what constitutes a "different word" (type) impact on the LD measures themselves and on their ability to discriminate between CEFR levels. The results show that basic measures of LD, such as the number of different words, the TTR (Templin 1957) and the Index of Guiraud (Guiraud 1954) explain more variance in the CEFR levels than sophisticated measures, such as D (Malvern et al. 2004), HD-D (McCarthy and Jarvis 2007) and MTLD (McCarthy 2005) provided text length is kept constant across texts. A simple count of different words (defined as lemma's and not as word families) was the best predictor of CEFR levels and explained 22 percent of the variance in overall scores on the Pearson Test of English Academic in essays written by 176 test takers.

show abstract

Section: Methodsmentioning

confidence: 99%

Back to Basics: How Measures of Lexical Diversity Can Help Discriminate between CEFR Levels

Treffers‐Daller¹,

Parslow

Williams

2016

Applied Linguistics

View full text Add to dashboard Cite

show abstract

“…Other lines of research have focused on differences between raters' perceptions of handwritten and typed texts, with typed essays receiving higher scores when controlling for language ability (Wolfe and Manalo, 2004;Breland, Lee, & Muraki, 2005). Essay length has also been shown to be highly correlated with measures of writing so that higher scores are associated with longer essays (Barkaoui, 2010;Kobrin, Deng, & Shaw, 2011). However, little attention has been directed toward the degree to which this effect is due to a true relationship between writing quality and essay length versus bias on the part of raters.…”

Section: Response Contentmentioning

confidence: 97%

Features of difficult-to-score essays

2016

View full text Add to dashboard Cite

“…It is well-known that in all timed impromptu essay tests writing length comprises a significant portion of the shared variance of the scores of human readers -even the College Board concedes that a significant portion of the score on the SAT writing section essay is attributable solely to the number of words (Beckman, 2010;Kobrin, Deng, & Shaw, 2007, 2011Winerip, 2012). Simply by overvaluing the number of words in an essay, AES machines can achieve correlations and shared variances that can, on first appearance and in some circumstances, match those achieved between two human readers.…”

Section: Fig 1 Average Shared Variance Between # Of Words and Scorementioning

confidence: 98%

When “the state of the art” is counting words

Perelman

2014

Assessing Writing

View full text Add to dashboard Cite

The association between SAT prompt characteristics, response features, and essay scores

Cited by 38 publications

References 23 publications

Back to Basics: How Measures of Lexical Diversity Can Help Discriminate between CEFR Levels

Back to Basics: How Measures of Lexical Diversity Can Help Discriminate between CEFR Levels

Features of difficult-to-score essays

When “the state of the art” is counting words

Contact Info

Product

Resources

About