Automated Trait Scores for TOEFL ® Writing Tasks

Educational Measurement

et al. 2021

Self Cite

Grouping individuals according to a set of measured characteristics, or profiling, is frequently used in describing, understanding, and acting on a phenomenon. The advent of computer‐based assessment offers new possibilities for profiling writing because aspects can be captured that were not heretofore observable. We explored whether writing processes could be profiled of over 30,000 adults taking a high‐school equivalency examination. Process features were extracted from keystroke logs, aggregated into composite indicators, and used with essay score to assign individuals to profiles. Analyses included computing the percentages of individuals that could be classified, using MANOVA to examine differences among profiles on external variables, and examining if profiles could be distinguished from one another based on patterns derived from cluster analysis. Results showed that about 30% of examinees could be classified into profiles that were largely distinct. These results contribute toward a foundation for using such profiles in describing how individuals compose and in how their writing might be improved.

Section: Discussionmentioning

confidence: 97%

mentioning

confidence: 99%

Are There Distinctive Profiles in Examinee Essay‐Writing Processes?

Bennett

Zhang

Educational Measurement

et al. 2021

Self Cite

Automated Trait Scores for TOEFL^® Writing Tasks

Attali

ETS Research Report Series

2015

Self Cite

The e‐rater® automated essay scoring system is used operationally in the scoring of TOEFL iBT® independent and integrated tasks. In this study we explored the psychometric added value of reporting four trait scores for each of these two tasks, beyond the total e‐rater score. The four trait scores are word choice, grammatical conventions, fluency and organization, and content. Trait scores were computed on the basis of several criteria for determining feature weights: regression parameters of the trait features on human scores, reliability of trait features, and coefficients of features from a principal component analysis. In addition, augmented trait scores, based on information from other trait scores, were also analyzed. The psychometric added value of trait scores beyond total e‐rater scores was evaluated by comparing the ability to predict a particular trait score on one task from the same trait score on the other task versus the e‐rater score on the other task. Results supported the use of trait scores, and are discussed in terms of their contribution to the construct validity of e‐rater as an alternative essay scoring method.

Automated Trait Scores for GRE^® Writing Tasks

Attali

ETS Research Report Series

2015

The e‐rater® automated essay scoring system is used operationally in the scoring of the argument and issue tasks that form the Analytical Writing measure of the GRE® General Test. For each of these tasks, this study explored the value added of reporting 4 trait scores for each of these 2 tasks over the total e‐rater score. The 4 trait scores are word choice, grammatical conventions, fluency and organization, and content. First, confirmatory factor analysis supported this underlying structure. Next, several alternative ways of determining feature weights for trait scores were compared: weights based on regression parameters of the trait features on human scores, reliability of trait features, and loadings of features from factor analytic results. In addition, augmented trait scores, based on information from other trait scores, were also analyzed. The added value of all trait score variants was evaluated by comparing the ability to predict a particular trait score on one task from either the same trait score on the other task or the e‐rater score on the other task. Results supported the use of trait scores and are discussed in terms of their contribution to the construct validity of e‐rater as an alternative essay scoring method.