Sample‐size Requirements for Automated Essay Scoring

Abstract

Sample‐size requirements were considered for automated essay scoring in cases in which the automated essay score estimates the score provided by a human rater. Analysis considered both cases in which an essay prompt is examined in isolation and those in which a family of essay prompts is studied. In typical cases in which content analysis is not employed and in which the only object is to score individual essays to provide feedback to the examinee, it appears that several hundred essays are sufficient. For application of one model to a family of essays, fewer than 100 essays per prompt may often be adequate. The cumulative logit model was explored as a possible replacement of the linear regression model usually employed in automated essay scoring; the cumulative logit model performed somewhat better than did the linear regression model.

Monitoring of Scoring Using the e‐rater® Automated Scoring System and Human Raters on a Writing Test

“…Davey () indicated that e‐rater and human scores are imperfectly related owing to this issue. Other models such as cumulative‐logit model (Haberman & Sinharay, ) should be explored and used for future research. Finally, human rating quality in the training sample: human scores need to be carefully monitored during the automated scoring process.…”

Comparison ofe-rater® Automated Essay Scoring Model Calibration Methods Based on Distributional Targets

“…Because the G and GOPSI models typically deal with a group of 12 or more prompts, they require smaller sample sizes per prompt than the PS models (Williamson, 2009). Less than 100 essays per prompt might be adequate for constructing a G model of a large family of prompts (Haberman & Sinharay, 2008). In contrast, the PS models require a greater volume of training essays.…”

“…Much effort has been made to develop e-rater scoring models that can be applied to produce reliable essay scores (e.g., Attali, 2007;Bridgeman, Trapani, & Attali, 2012;Haberman & Sinharay, 2008). To date, there are primarily three types of scoring models: the Generic Model (G Model), the Generic with Operational Prompt Specific Intercept Model (GOPSI Model), and the Prompt Specific Model (PS Model).…”

INVESTIGATING THE SUITABILITY OF IMPLEMENTING THE E‐RATER® SCORING ENGINE IN A LARGE‐SCALE ENGLISH LANGUAGE TESTING PROGRAM

“…Traditionally, e-rater models are built on a subset of the essays and evaluated on the remaining essays. Researchers had also proposed to construct a scoring model in a jackknife or n-fold cross-validation fashion (Haberman & Sinharay, 2008). 2 This way, not only the sample size for model calibration is largely increased, but the variation in the entire data set can also be reflected in both model calibration and evaluation.…”

“…PS (press) models were built for each of the 63 prompts separately applying a leave-oneout sampling approach. Models were evaluated using indices derived from the PRESS statistic (Kutner, Nachtsheim, Neter, & Li, 2005, p. 360 Guilford & Fruchter, 1973;Haberman & Sinharay, 2008;and Weisberg, 1985, for exceptions. ) In this study, we used a total of four PRESS-derived indices.…”