Process Features in Writing: Internal Structure and Incremental Value Over Product Features

Abstract

In educational measurement contexts, essays have been evaluated and formative feedback has been given based on the end product. In this study, we used a large sample collected from middle school students in the United States to investigate the factor structure of the writing process features gathered from keystroke logs and the association of that latent structure with the quality of the final product (i.e., the essay text). The extent to which those process factors had incremental value over product features was also examined. We extracted 29 process features using the keystroke logging engine developed at Educational Testing Service (ETS). We identified 4 factors that represent the extent of writing fluency, local word‐level editing, phrasal/chunk‐level editing, and planning and deliberation during writing. We found that 2 of the 4 factors—writing fluency, and planning and deliberation—significantly related to the quality of the final text, whereas the 4 factors altogether accounted for limited variance in human scores. In 1 of the 2 samples studied, the keystroke‐logging fluency factor added incrementally, but only marginally, to the prediction of human ratings of text‐production skills beyond product features. The limited power of the writing process features for predicting human scores and the lack of clear additional predictive value over product features are not surprising given that the human raters have no knowledge of the writing process leading to the final text and that the product features measure the basic text quality specified in the human scoring rubric. Study limitations and recommendation for future research are also provided.

Writing Processes in Short Written Responses to Questions Probing Prior Knowledge

“…Among weaker writers, text tends to be produced less efficiently and pauses appear in locations that suggest difficulties in typing, spelling, word finding, and other transcription processes. We have obtained similar results in analyses of keystroke patterns among middle school students completing various essay tasks (Deane, ; Deane & Zhang, ; Zhang & Deane, ).…”

“…Many of the features identified in this way were in the initial set of features described by Almond, Deane, Quinlan, and Wagner (). The features could be defined meaningfully without missing data, even for behaviors like editing, in which the data were very sparse. We thus favored features for which we could provide meaningful default values for feature calculation. If possible, the feature was related to one of the factors we identified in previous work (Deane, ; Deane & Zhang, ; Zhang & Deane, ), in particular, planning and deliberation, fluency (or keyboarding effort), and effort put into local editing or revision. However, given the first two constraints, we did not use feature sets identical to those reported in Deane (), Deane and Zhang (), or Zhang and Deane (). Variables were transformed to approximate a normal distribution when possible.…”

“…We thus favored features for which we could provide meaningful default values for feature calculation. If possible, the feature was related to one of the factors we identified in previous work (Deane, ; Deane & Zhang, ; Zhang & Deane, ), in particular, planning and deliberation, fluency (or keyboarding effort), and effort put into local editing or revision. However, given the first two constraints, we did not use feature sets identical to those reported in Deane (), Deane and Zhang (), or Zhang and Deane (). Variables were transformed to approximate a normal distribution when possible. To this end, count features were transformed by taking the square root and duration and relative probability features by taking the logarithm.…”

“…The studies we cite share one striking feature: Nearly all of the features that correlate with writing quality can be interpreted in terms of a simple, knowledge‐telling approach to writing in which students generate ideas and then write them down in a simple, sequential fashion without recourse to complex revision or editing strategies. Deane () and Zhang and Deane () observed that the strongest associations with essay scores are factors that appear to measure general fluency and sentence‐level planning and deliberation. Deane and Zhang () observed that the best predictors of writing quality are total time on task, length of bursts of text production, duration of pauses within and between words, and (with relatively low weight) the duration of simple editing events, such as cutting/pasting and backspacing.…”

Advancing Human Assessment: A Synthesis Over Seven Decades

“…However, important new lines of work were added. One such line, made possible by computer-based assessment, was the analysis of keystroke logs generated by students as they responded to essays, simulations, and other performance tasks von Davier 2015, 2016;Zhang and Deane 2015). This analysis began to open a window into the processes used by students in problem solving.…”