Automated Scoring Within a Developmental, Cognitive Model of Writing Proficiency

Abstract

ETS has recently instituted the Cognitively Based Assessments of, for, and as Learning (CBAL) research initiative to create a new generation of assessment designed from the ground up to enhance learning. It is intended as a general approach, covering multiple subject areas including reading, writing, and math. This paper is concerned with the writing assessment being developed within the CBAL framework, and in particular, with the potential for using automated scoring techniques effectively within such an assessment to support learning.Key words: writing assessment, CBAL, automated scoring ii (2008) Context and PurposeIn practice, assessment design involves complex tradeoffs. For instance, many testing programs assess writing directly by administering generic prompts under strict time limits. This type of assessment has gained in popularity relative to indirect measures of writing skill in response to the concern that writing should be assessed by actually requiring students to write, but it has been criticized as encouraging simplistic, formulaic approaches to instruction that fail to develop the complex skills students need to learn in order to become effective writers (Hillocks, 1987(Hillocks, , 2002. Portfolio-based writing assessment may encourage richer, more appropriate forms of instruction, but it raises issues of reliability and validity. Automated scoring methods have been introduced and advanced as instructional aids, but they have been criticized as focusing attention too much on mechanical correction of errors rather than encouraging critical engagement with content. In other words, it is no simple matter to design summative assessments that are truly tests worth teaching to (Shepard, 2002). Success in creating tests that truly encourage learning will require a complex, sustained effort that takes into account an extraordinary range of elements, including at least the following:• the cognitive and instructional literatures that define what students learn and how they best learn it• psychometric constraints on effective assessment• institutional and public policy constraints that ultimately determine test feasibility ETS has recently instituted a research initiative, Cognitively Based Assessments of, for, and as Learning (CBAL), intended to address this problem and create a new generation of assessment designed from the ground up to enhance learning. It is intended as a general approach, covering multiple subject areas including reading, writing, and math (see Bennett & Gitomer, 2009). This paper is concerned with the writing assessment being developed within the CBAL framework, and in particular, with the potential for using automated scoring techniques effectively within such an assessment to support learning. The role of automated scoring for CBAL cannot be discussed, however, without an in-depth presentation of the CBAL initiative.This presentation occupies section 1.2, after which section 1.3 examines the role of automated scoring within this philosophy. The remaining sections of th...

Exploring the Feasibility of Using Writing Process Features to Assess Text Production Skills

“…Several broad trends have been observed relating pause patterns during writing with the quality of the resulting text (Alves, Branco, Castro, & Olive, ; Connelly, Dockrell, Walter, & Critten, ; Hayes, ; Kaufer, Hayes, & Flower, ; Miller, ; van den Bergh & Rijlaarsdam, ; Wengelin, ). Similar results have been obtained in prior published research at Educational Testing Service (ETS), including studies by Almond, Deane, Quinlan, Wagner, and Sydorenko (), Deane and Quinlan (), and Deane, Quinlan, and Kostin (). In particular, for stronger writers, text tends to be produced efficiently in longer bursts; pauses are more likely to happen at natural loci for planning such as clause and sentence boundaries, and more editing and revision behavior can be observed.…”

“…This study purposefully did not examine the relationship between process/keystroke timing features and the product features used in automated essay scoring systems. These issues have been explored elsewhere (e.g., Deane, 2014;Deane, Quinlan, & Kostin, 2011), and we expect to explore them in greater depth in future publications (e.g., Zhang & Deane, in press). Our immediate goal in this study was to examine the extent to which writing process features provide measurement of fundamental text production skills, without reference to features of the final written product.…”

“…Statistical methods include calculation of summary features, such as burst length and pause duration, with some work using mixture models and principal component analysis (Almond et al, 2012;Baaijen, Galbraith, & de Glopper, 2012;Deane, 2014). Deane and Quinlan (2010), and Deane, Quinlan, & Kostin (2011) explored ways in which writing timing and process features can function as predictors of writing scores, but Deane (2014) found that issues of stability and generalization might exist, at least for some keystroke features. In particular, Deane (2014) examined the relationship between keystroke features from essays instantiating very different genres and topics and found relatively low correlations of keystroke features across paired student essays addressing different topics and genres combined with somewhat different patterns of correlations with external variables.…”

A Preliminary Analysis of Keystroke Log Data From a Timed Writing Task

“…The CBAL™ writing (Deane, 2011(Deane, , 2012Deane, Fowles, Baldwin, & Persky, 2011;Deane, Quinlan, & Kostin, 2011) assessment features a variety of writing a nd literacy tasks, culminating in a short essay. The short essay is thought to be representative of other more complex writing tasks that students will face later in academic or professional life.…”

“…Finally, to take a quick look to see the value of timing features in automated scoring, we examine some regression models to predict the human scores from automated features. In this analysis, we used three sets of features: the eight generic e-rater features (Attali & Burstein, 2006), eight factors arising from the Source Finder features identified by Deane, Quinlan, and Kostin (2011), and the timing features described above. We used the R (R Development Core Team, 2009) step function to search for the model with the lowest BIC 8 statistic.…”