Evaluations of Automated Scoring Systems in Practice

Rotou, Ourania; Rupp, André

doi:10.1002/ets2.12293

Cited by 7 publications

(6 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As we see an increasing number of studies that explore applying or extending ML methods to solving measurement problems, we hope to call the attention of the measurement community to identify, discuss, and work together to address the risks. This concern is shared by some of the articles we reviewed (e.g., Alexander et al, 2020;Anderson et al, 2020;Higgins & Heilman, 2014;Rauthmann, 2020;Rotou & Rupp, 2020;Stachl et al, 2020).…”

Section: Risks Associated With Involving ML In Measurementmentioning

confidence: 95%

“…The other four articles include examples of new processes for validating ML-enhanced measurement. Lottridge et al (2020) and Rotou and Rupp (2020) suggested practical processes for evaluating automated scoring systems. Speer (2021) investigated the validity and generalizability evidence for scores generated using natural language processing (NLP) algorithms.…”

Section: Risks Associated With Involving ML In Measurementmentioning

confidence: 99%

See 1 more Smart Citation

MxML (Exploring the Relationship between Measurement and Machine Learning): Current State of the Field

Zheng,

Nydick,

Huang

et al. 2024

Educational Measurement

View full text Add to dashboard Cite

The recent surge of machine learning (ML) has impacted many disciplines, including educational and psychological measurement (hereafter shortened as measurement). The measurement literature has seen rapid growth in applications of ML to solve measurement problems. However, as we emphasize in this article, it is imperative to critically examine the potential risks associated with involving ML in measurement. The MxML project aims to explore the relationship between measurement and ML, so as to identify and address the risks and better harness the power of ML to serve measurement missions. This paper describes the first study of the MxML project, in which we summarize the state of the field of applications, extensions, and discussions about ML in measurement contexts with a systematic review of the recent 10 years’ literature. We provide a snapshot of the literature in (1) areas of measurement where ML is discussed, (2) types of articles (e.g., applications, conceptual, etc.), (3) ML methods discussed, and (4) potential risks associated with involving ML in measurement, which result from the differences between what measurement tasks need versus what ML techniques can provide.

show abstract

Section: Risks Associated With Involving ML In Measurementmentioning

confidence: 95%

Section: Risks Associated With Involving ML In Measurementmentioning

confidence: 99%

MxML (Exploring the Relationship between Measurement and Machine Learning): Current State of the Field

Zheng,

Nydick,

Huang

et al. 2024

Educational Measurement

View full text Add to dashboard Cite

show abstract

“…As we see an increasing number of studies that explore applying or extending ML methods to solving measurement problems, we hope to call the attention of the measurement community to identify, discuss, and work together to address the risks. This concern is shared by some of the articles we reviewed (e.g., Alexander III et al, 2020;Anderson et al, 2020;Higgins & Heilman, 2014;Rauthmann, 2020;Rotou & Rupp, 2020;Stachl et al, 2020).…”

Section: Risks Associated With Involving ML In Measurementmentioning

confidence: 95%

Section: Risks Associated With Involving ML In Measurementmentioning

confidence: 99%

MxML (Exploring the relationship between measurement and machine learning): Current state of the field

Zheng¹,

Nydick²,

Huang³

et al. 2023

Preprint

View full text Add to dashboard Cite

The recent surge of machine learning (ML) has impacted many disciplines, including educational and psychological measurement (hereafter shortened as measurement, “M”). The measurement literature has seen a rapid growth in studies that explore using ML methods to solve measurement problems. However, there exist gaps between the typical paradigm of ML and fundamental principles of measurement. The MxML project was created to explore how the measurement community might potentially redefine the psychometrics discipline in the imminent future of big data and machine learning, so as to harness the power of machine learning to serve our (redefined and updated) mission. This paper describes the first study of the MxML project, in which we summarize the state of the field of applications, extensions, and discussions about ML methods in measurement contexts with a systematic review of the recent 10 years of literature (2013 - 2022). Specifically, we provide a snapshot of the literature in terms of (1) areas of measurement, (2) types of article, (3) ML methods discussed, and (4) gaps addressed between measurement goals and ML methods.

show abstract

“…As substitutes for human raters, AES systems utilize computer technology to evaluate and score written prose. At the very minimum, AES systems provide consistency in scoring essays, and are not time-consuming for scoring routine, written-language assignments (Ifenthaler & Dikli, 2015; Rotou & Rupp, 2020).…”

Section: Introductionmentioning

confidence: 99%

Application of an Automated Essay Scoring engine to English writing assessment using Many-Facet Rasch Measurement

Chan¹,

Bond

Yan

2022

Language Testing

View full text Add to dashboard Cite

We investigated the relationship between the scores assigned by an Automated Essay Scoring (AES) system, the Intelligent Essay Assessor (IEA), and grades allocated by trained, professional human raters to English essay writing by instigating two procedures novel to written-language assessment: the logistic transformation of AES raw scores into hierarchically ordered grades, and the co-calibration of all essay scoring data in a single Rasch measurement framework. A total of 3453 essays were written by 589 US students (in Grades 4, 6, 8, 10, and 12), in response to 18 National Assessment of Educational Progress (NAEP) writing prompts at three grade levels (4, 8, & 12). We randomly assigned one of two versions of the assessment, A or B, to each student. Each version comprised a narrative (N), an informative (I), and a persuasive (P) prompt. Nineteen experienced assessors graded the essays holistically using NAEP scoring guidelines, using a rotating plan in which each essay was rated by four raters. Each essay was additionally scored using the IEA. We estimated the effects of rater, prompt, student, and rubric by using a Many-Facet Rasch Measurement (MFRM) model. Last, within a single Rasch measurement scale, we co-calibrated the students’ grades from human raters and their grades from the IEA to compare them. The AES machine maintained equivalence with human scored ratings and were more consistent than those from human raters. 多面Rasch測量(MFRM)模型應用於自動文章評分(AES)系統作英文作文評估

show abstract

Evaluations of Automated Scoring Systems in Practice

Cited by 7 publications

References 24 publications

MxML (Exploring the Relationship between Measurement and Machine Learning): Current State of the Field

MxML (Exploring the Relationship between Measurement and Machine Learning): Current State of the Field

MxML (Exploring the relationship between measurement and machine learning): Current state of the field

Application of an Automated Essay Scoring engine to English writing assessment using Many-Facet Rasch Measurement

Contact Info

Product

Resources

About