A Preliminary Analysis of Keystroke Log Data From a Timed Writing Task

Zhang

2015

Self Cite

In this report, we examine the feasibility of characterizing writing performance using process features derived from a keystroke log. Using data derived from a set of CBAL™ writing assessments, we examine the following research questions: (a) How stable are the keystroke timing and process features across testing occasions? (b) How consistent are the patterns of feature–human correlation across genres and topics? (c) How accurately can we predict human ratings on writing fundamentals using a combination of the keystroke timing and process features, and what are the contributions of each feature to the reliable variance in the human ratings? (d) If we train a predictive model on one prompt, how well do its predictions generalize to the other prompts of the same or different genre? The results of the study indicate that keystroke log features vary considerably in stability across testing occasions and display somewhat different patterns of feature–human correlation across genres and topics. However, using the most stable features, we can obtain moderate to strong prediction of human essay scores, and those models generalize reasonably well across prompts though more strongly within than across writing genres.

supporting

confidence: 89%

Section: Methodsmentioning

confidence: 99%

mentioning

confidence: 99%

See 1 more Smart Citation

Exploring the Feasibility of Using Writing Process Features to Assess Text Production Skills

Zhang

2015

Self Cite

“…The keystroke logging data were analyzed to identify several key event types previously identified from the writing literature (Almond, Deane, Quinlan, & Wagner, ) using a segmentation program that identified word and sentence boundaries from local cues, such as white space and punctuation marks. The analysis identified bursts (sequences of keystrokes without long pauses), cut/paste/jump events, backspacing events, and several types of pauses (between characters within a word, between words, and between sentences, among others).…”

Section: Methodsmentioning

confidence: 99%

Using Writing Process and Product Features to Assess Writing Quality and Explore How Those Features Relate to Other Literacy Tasks

2014

Self Cite

This paper explores automated methods for measuring features of student writing and determining their relationship to writing quality and other features of literacy, such as reading rest scores. In particular, it uses the e‐rater® automatic essay scoring system to measure product features (measurable traits of the final written text) and features extracted from keystroke logs to measure process features (measurable features of the writing process). These techniques are applied to student essays written during large‐scale pilot administrations of writing assessments developed for ETS's CBAL™ research initiative. The design makes it possible to explore the factor structures of these product and process features and to examine how well they generalize beyond a single test session to predict underlying traits such as writing ability and reading level. Three product factors are identified, connected to fluency, accuracy, and content. The process factors are identified, corresponding to hesitancy behaviors, editing behaviors, and burst span (the extent to which text is produced in long bursts with only short internal pauses). The results suggest that writing process and product features have stable factor structures that generalize to predict writing quality, though there are some genre‐ or task‐specific differences.

“…Where possible, we selected features where there was a significant correlation greater than .2 between the feature value obtained for this feature in the word listing item and the corresponding feature value for the knowledge elicitation item in at least one of the four forms. Many of the features identified in this way were in the initial set of features described by Almond, Deane, Quinlan, and Wagner (). The features could be defined meaningfully without missing data, even for behaviors like editing, in which the data were very sparse. We thus favored features for which we could provide meaningful default values for feature calculation. If possible, the feature was related to one of the factors we identified in previous work (Deane, ; Deane & Zhang, ; Zhang & Deane, ), in particular, planning and deliberation, fluency (or keyboarding effort), and effort put into local editing or revision.…”

Section: Methodsmentioning

confidence: 99%

Writing Processes in Short Written Responses to Questions Probing Prior Knowledge

O’Reilly

Chao

et al. 2018

Self Cite

The purpose of the report is to explore some of the mechanisms involved in the writing process. In particular, we examine students' process data (keystroke log analysis) to uncover how students approach a knowledge‐telling task using 2 different task types. In the first task, students were asked to list as many words as possible related to a particular topic (word listing). In a second task, students were asked to write to a specific prompt that was designed to elicit their background knowledge of a topic using connected text (knowledge elicitation). Using a matrix incomplete block design, 1,592 high school students completed the 2 writing tasks in addition to a multiple‐choice test of their background knowledge in 2 of 5 possible topics in the domain of U.S. history. An array of process data including students' typing and associated timing features was used to predict the writing scores on the 2 different types of tasks. The analyses revealed several distinct patterns that were associated with processing at the task knowledge productivity level, the editing effort level, and the keyboarding effort level. The robustness of the features was reflected in a set of hierarchal regressions that demonstrated that the process features were predictive of the writing score even when students' knowledge scores on the associated multiple‐choice test were considered. In sum, the results indicate that process data in the form of log file analysis are useful for both understanding the writing process and exploring potential differences between students with high and low knowledge.