A Study of the Use of the e‐rater^® Scoring Engine for the Analytical Writing Measure of the GRE^® revised General Test

Abstract

In this research, we investigated the feasibility of implementing the e‐rater® scoring engine as a check score in place of all‐human scoring for the Graduate Record Examinations® (GRE®) revised General Test (rGRE) Analytical Writing measure. This report provides the scientific basis for the use of e‐rater as a check score in operational practice. We proceeded with the investigation in four phases. In phase I, for both argument and issue prompts, we investigated the quality of human scoring consistency across individual prompts, as well as two groups of prompts organized into sets. The sets were composed of prompts with separate focused questions (i.e., variants) that must be addressed by the writer in the process of responding to the topic of the prompt. There are also groups of variants of prompts (i.e., grouped for scoring purposes by similar variants). Results showed adequate human scoring quality for model building and evaluation. In phase II, we investigated eight different e‐rater model variations each for argument and issue essays including prompt‐specific; variant‐specific; variant‐group–specific; and generic models both with and without content features at the rating level, at the task score level, and at the writing score level. Results showed the generic model was a valued alternative to the prompt‐specific, variant‐specific, and variant‐group–specific models, with and without the content features. In phase III, we evaluated the e‐rater models on a recently tested group from the spring of 2012 (between March 18, 2012, to June 18, 2012) following the introduction of scoring benchmarks. Results confirmed the feasibility of using a generic model at the rating and task score level and at the writing score level, demonstrating reliable cross‐task correlations, as well as divergent and convergent validity. In phase IV of the study, we purposely introduced a bias to simulate the effects of training the model on a potentially less able group of test takers in the spring of 2012. Results showed that use of the check‐score model increased the need for adjudications between 5% and 8%, yet the increase in bias actually increased the agreement of the scores at the analytical writing score level with all‐human scoring.

Implementing a Contributory Scoring Approach for the GRE® Analytical Writing Section: A Comprehensive Empirical Investigation

“…In contrast, the primary adjudication threshold has been set at 1.5 for both the TOEFL independent task and the PRAXIS ® test's argumentative task under a contributory scoring approach. Similarly, it was recently reset to 2.0 for the TOEFL integrated task, with the potentially undesirable effects of the larger primary threshold on score separation compensated for by giving human ratings twice the weight of the machine scores for reporting purposes (Breyer et al, 2014;Ramineni, Trapani, & Williamson, 2015;Ramineni, Trapani, Williamson, Davey, & Bridgeman, 2012b;Ramineni et al, 2012a).…”

“…The argument of critics is that “nonsense” and perhaps “obviously flawed” essays that result from gaming attempts can be detected by human readers but not always by built‐in machine detectors (i.e., advisories) in the automated scoring system (see Ramineni, Trapani, Williamson, Davey, & Bridgeman, , or Breyer et al, , for a description of the different advisories evaluated for the GRE‐AW section). For the purpose of this report, the sole machine scoring approach is not considered further for the GRE‐AW section because the consequential use of the GRE‐AW section scores is associated with relatively high stakes for individual test takers.…”

“…A major consideration for determining the primary adjudication threshold through empirical evidence has been the observation that human and machine scores can separate from each other for some—but not all—subgroups of test takers that are pertinent in comprehensive fairness evaluations; we call this issue score separation in short (Breyer et al, ; Bridgeman, Trapani, & Attali, ). Score separation is not something that can be evaluated properly by focusing on human–machine score differences at the rating level but, rather, needs to be evaluated by focusing on these differences at the task or reported scale score level once adjudication procedures have been applied.…”

“…This methodological step is about determining the weights that the human ratings and, possibly, machine scores receive under a particular scoring approach. Common past practice at ETS has been to equally weight any ratings when creating task scores (Breyer et al, ; Ramineni et al, ). However, as noted, there are exceptions.…”

Evaluation of e‐rater® for the Praxis I® Writing Test

“…The automated scoring models based on e‐rater have been successfully evaluated in recent years for the writing prompts included in the old GRE General Test (Ramineni, Trapani, Williamson, Davey, & Bridgeman, ), the TOEFL iBT ® test (Attali, Bridgeman, & Trapani, ; Ramineni, Trapani, Williamson, Davey, & Bridgeman, ), and the GRE revised General Test (Breyer et al, ). The current TOEFL iBT test uses e‐rater for operational scoring of the essay tasks, and the GRE uses e‐rater as a quality control on the reported human scores, thus allowing the programs to report scores efficiently and to use their human rater pool more effectively.…”