Exploring the Feasibility of Using Writing Process Features to Assess Text Production Skills

Abstract

In this report, we examine the feasibility of characterizing writing performance using process features derived from a keystroke log. Using data derived from a set of CBAL™ writing assessments, we examine the following research questions: (a) How stable are the keystroke timing and process features across testing occasions? (b) How consistent are the patterns of feature–human correlation across genres and topics? (c) How accurately can we predict human ratings on writing fundamentals using a combination of the keystroke timing and process features, and what are the contributions of each feature to the reliable variance in the human ratings? (d) If we train a predictive model on one prompt, how well do its predictions generalize to the other prompts of the same or different genre? The results of the study indicate that keystroke log features vary considerably in stability across testing occasions and display somewhat different patterns of feature–human correlation across genres and topics. However, using the most stable features, we can obtain moderate to strong prediction of human essay scores, and those models generalize reasonably well across prompts though more strongly within than across writing genres.

Writing Processes in Short Written Responses to Questions Probing Prior Knowledge

“…Among weaker writers, text tends to be produced less efficiently and pauses appear in locations that suggest difficulties in typing, spelling, word finding, and other transcription processes. We have obtained similar results in analyses of keystroke patterns among middle school students completing various essay tasks (Deane, ; Deane & Zhang, ; Zhang & Deane, ).…”

“…If we compare these results to the results of our prior studies of middle school essay writing (i.e., Deane, ; Deane & Zhang, ; Zhang & Deane, ), there appear to be strong similarities in the feature patterns associated with high and low performance. In both the present and in prior studies, the bulk of the variance in student scores can be accounted for using measures of planning and productivity (including total time, burst lengths, time spent before typing begins, and pauses between words).…”

“…Many of the features identified in this way were in the initial set of features described by Almond, Deane, Quinlan, and Wagner (). The features could be defined meaningfully without missing data, even for behaviors like editing, in which the data were very sparse. We thus favored features for which we could provide meaningful default values for feature calculation. If possible, the feature was related to one of the factors we identified in previous work (Deane, ; Deane & Zhang, ; Zhang & Deane, ), in particular, planning and deliberation, fluency (or keyboarding effort), and effort put into local editing or revision. However, given the first two constraints, we did not use feature sets identical to those reported in Deane (), Deane and Zhang (), or Zhang and Deane (). Variables were transformed to approximate a normal distribution when possible.…”

“…We thus favored features for which we could provide meaningful default values for feature calculation. If possible, the feature was related to one of the factors we identified in previous work (Deane, ; Deane & Zhang, ; Zhang & Deane, ), in particular, planning and deliberation, fluency (or keyboarding effort), and effort put into local editing or revision. However, given the first two constraints, we did not use feature sets identical to those reported in Deane (), Deane and Zhang (), or Zhang and Deane (). Variables were transformed to approximate a normal distribution when possible. To this end, count features were transformed by taking the square root and duration and relative probability features by taking the logarithm.…”

“…Deane () and Zhang and Deane () observed that the strongest associations with essay scores are factors that appear to measure general fluency and sentence‐level planning and deliberation. Deane and Zhang () observed that the best predictors of writing quality are total time on task, length of bursts of text production, duration of pauses within and between words, and (with relatively low weight) the duration of simple editing events, such as cutting/pasting and backspacing. Except for the editing events, these are features that measure the fluency with which writers can generate ideas, translate them into words, and output them using a keyboard.…”