Automated Essay Scoring With E‐rater® V.2.0

Attali, Yigal; Burstein, Jill

doi:10.1002/j.2333-8504.2004.tb01972.x

Cited by 399 publications

(594 citation statements)

References 12 publications

Supporting

Mentioning

572

Contrasting

Unclassified

Order By: Relevance

“…In early 2006 the users of TPO could instantly receive scores on reading and listening sections, both based on multiple-choice items, as well as the writing section, with automated writing scores provided by e-rater ® (Attali & Burstein, 2006). The scores on speaking sections were produced by human raters within 5 business days.…”

Section: The Tpo Assessmentmentioning

confidence: 99%

“…al., 1998;Clauser, Margolis, Clyman, & Ross, 1997;Williamson, Bejar, & Hone, 1999) has the ability for computerized delivery and automated scoring of constructed-response items enabled the practical operational use of automated scoring for such items. Initially, such applications were primarily in automated scoring of essays (e.g., Attali & Burstein, 2006;Burstein et al, 1998;Chodorow & Burstein, 2004;Landauer & Dumais, 1997;Rudner, Garcia, & Welch, 2006), which has matured to a considerable degree. However, recent research in natural language processing and speech recognition capabilities has expanded the nature of constructed-response tasks that are automatically scorable to include short-answer tasks requiring factual information (e.g., Leacock & Chodorow, 2003) and tasks eliciting highly predictable speech (e.g., Bernstein, 1999).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

AUTOMATED SCORING OF SPONTANEOUS SPEECH USING SPEECHRATER^SM V1.0

Xi¹,

Higgins²,

Zechner³

et al. 2008

ETS Research Report Series

View full text Add to dashboard Cite

This report presents the results of a research and development effort for SpeechRater SM Version 1.0 (v1.0), an automated scoring system for the spontaneous speech of English language learners used operationally in the Test of English as a Foreign Language™ (TOEFL ® ) Practice Online assessment (TPO). The report includes a summary of the validity considerations and analyses that drive both the development and the evaluation of the quality of automated scoring. These considerations include perspectives on the construct of interest, the context of use, and the empirical performance of the SpeechRater in relation to both the human scores and the intended use of the scores. The outcomes of this work have implications for short-and long-term goals for iterative improvements to SpeechRater scoring. which is used by prospective test takers to prepare for the official TOEFL iBT test. This study reports the development and validation of the system for low-stakes practice purposes. The process we followed to build this system represented a principled approach to maximizing 2 essential qualities: substantively meaningful and technically sound. In developing and evaluating the features and the scoring models to predict human assigned scores, we engaged both content and technical experts actively to ensure the construct representation and technical soundness of the system. We compared primarily two alternative methodologies of building scoring modelsmultiple regression and classification trees-in terms of their construct representation and empirical performance in predicting human scores. Based on the evaluation results, we concluded that a multiple regression model with feature weights determined by content experts was superior to the other competing models evaluated.We then used an argument-based approach to integrate and evaluate the existing evidence to support the use of SpeechRater SM v1.0 in a low-stakes practice environment. The argumentbased approach provided a mechanism for us to articulate the strengths and weaknesses in the validity argument for using SpeechRater v1.0 and put forward a transparent argument for using it for a low-stakes practice environment. In particular, the construct representation of the multiple regression model with expert weights was sufficiently broad to justify its use in a low-stakes application. While some higher-order aspects of the speaking construct (such as content and organization) are missing, more basic aspects of the construct (such as pronunciation and fluency) are richly represented. In addition, these different parts of the speaking construct tend to be highly correlated, so that the absence of higher order factors is not as detrimental to the model's agreement with human raters as it otherwise might be. The model's agreement with human raters was not sufficiently high to support high-stakes decisions but was still suitable for use in low-stakes applications. The correlation of the 6-item aggregate score with human raters was .57 and was deemed acceptable given the lo...

show abstract

Section: The Tpo Assessmentmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

AUTOMATED SCORING OF SPONTANEOUS SPEECH USING SPEECHRATER^SM V1.0

Xi¹,

Higgins²,

Zechner³

et al. 2008

ETS Research Report Series

View full text Add to dashboard Cite

show abstract

“…essays (Burstein et al 1998). The system has continued to evolve (Attali and Burstein 2006;Burstein et al 2004;Burstein et al 2013) and has become a major ETS asset. Importantly, the inner workings of e-rater are well documented (Attali and Burstein 2006;Quinlan et al 2009), and disclosed through patents.…”

Section: Automated Scoringmentioning

confidence: 99%

A Historical Survey of Research Regarding Constructed-Response Formats

Bejar

2017

Methodology of Educational Measurement and Assessment

View full text Add to dashboard Cite

This chapter chronicles ETS research and development contributions related to the use of constructed-response item formats. 1 The use of constructed responses in testing dates back to imperial China, where tests were used in the selection of civil servants. However, in the United States, the multiple-choice format became dominant during the twentieth century, following its invention and use by the SAT ® examinations created by the College Board in 1926. When ETS was created in 1947, post-secondary admissions testing was largely based on tests consisting of multiplechoice items. However, from the start, there were two camps at ETS: those who believed that multiple-choice tests were sufficiently adequate for the purpose of assessing "verbal" skills and those who believed that "direct" forms of assessment requiring written responses had a role to play. For constructed-response formats to regain a foothold in American education several hurdles would need to be overcome. Research at ETS was instrumental in overcoming those hurdles.The first hurdle was that of reliability, specifically the perennial issue of low interrater agreement, which plagued the acceptance of constructed-response formats for most of the twentieth century. The second hurdle was broadening the conception of validity to encompass more than predictive considerations, a process that began with the introduction of construct validity by Cronbach and Meehl (1955). Samuel Messick at ETS played a crucial role in this process by making construct validity relevant to educational tests. An inflexion point in the process of reincorporating constructed-response formats more widely in educational tests was marked 1 Constructed responses to a prompt or question can range in scope and complexity. Perhaps the most common constructed response is the written essay. However, short written responses to questions are also considered to be constructed, as are spoken answers in response to a prompt, mathematical responses (equations, plotted functions, etc.), computer programs, and graphical responses such as architectural designs.

show abstract

“…The main advantages of MR, on the other hand, are its much simpler design and much longer history of being used for automated scoring purposes (such as, e.g., in erater® (Attali and Burstein, 2005)). The relationships between the features and the predicted scores are straightforward, as well as the relative weights of the features, and so it may be more perspicuous to an outsider than a CART model.…”

Section: Scoring Modelsmentioning

confidence: 99%

“…Initially, such applications were primarily in automated scoring of essays (e.g. Burstein et al, 1998;Chodorow & Burstein, 2004;Attali & Burstein, 2005;Landauer, & Dumais, 1997;Rudner et al, 2006), which has matured to a considerable degree. However, recent research in natural language processing and speech recognition capabilities has expanded the nature of constructed response tasks that are automatically scorable to include short answer tasks requiring factual information (e.g., Leacock, 2004;Leacock & Chodorow, 2003) and tasks eliciting highly predictable speech (e.g., Bernstein, 1999).…”

Section: Introductionmentioning

confidence: 99%