Monitoring of Scoring Using the <i>e‐rater</i><sup>®</sup> Automated Scoring System and Human Raters on a Writing Test

Wang, Zhen; Davier, Alina A. von

doi:10.1002/ets2.12005

Cited by 13 publications

(19 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…It follows that a fair automated scoring system should not introduce additional construct-irrelevant group-related variance or disadvantage any group of test-takers in comparison to human scores (Penfield, 2016). Several standard measures have been used to evaluate the fairness of the automated scoring systems across different groups, for example speakers of different languages or test-takers with disabilities (Burstein and Chodorow, 1999;Bridgeman et al, 2012;Wang and von Davier, 2014;Wang et al, 2016;Loukina and Buzick, 2017). The two most common analyses are standardized mean score differences and overall model performance for different groups with human scores (predictive ability) (Ramineni and Williamson, 2013;Williamson et al, 2012).…”

Section: Fairness Metrics For Automated Scoringmentioning

confidence: 99%

The many dimensions of algorithmic fairness in educational applications

Loukina

Madnani

Zechner

2019

Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications

View full text Add to dashboard Cite

The issues of algorithmic fairness and bias have recently featured prominently in many publications highlighting the fact that training the algorithms for maximum performance may often result in predictions that are biased against various groups. Educational applications based on NLP and speech processing technologies often combine multiple complex machine learning algorithms and are thus vulnerable to the same sources of bias as other machine learning systems. Yet such systems can have high impact on people's lives especially when deployed as part of highstakes tests. In this paper we discuss different definitions of fairness and possible ways to apply them to educational applications. We then use simulated and real data to consider how test-takers' native language backgrounds can affect their automated scores on an English language proficiency assessment. We illustrate that total fairness may not be achievable and that different definitions of fairness may require different solutions.

show abstract

Section: Fairness Metrics For Automated Scoringmentioning

confidence: 99%

The many dimensions of algorithmic fairness in educational applications

Loukina

Madnani

Zechner

2019

Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications

View full text Add to dashboard Cite

show abstract

“…För engelska texter finns redan välutvecklade verktyg för automatisk textanalys, till exempel Coh-metrix som är fritt tillgängligt på internet (McNamara et al, 2014) och kommersiellt framtagna bedömningsverktyg som E-rater (Monaghan & Bridgeman, 2005;Wang & Davier, 2014). För svenska texter har automatisk bedömning endast tillämpats i ett fåtal fall och då med ett begränsat antal lingvistiska mått för den automatiska analysen (Östling et al, 2013;Kann, 2013).…”

Section: Automatisk Bedömning Av Elevtexterunclassified

“…Textegenskaper av mer kvalitativ natur, som originalitet i textens tankeinnehåll eller textens kommunikativa röst, kan inte fångas av en dator. Det är också i kombination med en "human rater" som verktyget "E-rater" används vid ETS i USA (Monaghan & Bridgeman, 2005;Wang & Davier, 2014).…”

Section: Automatisk Bedömning Av Elevtexterunclassified

Kvantiteter i kvalitativt bedömda elevtexter – framtida verktyg för rättvis bedömning?

Palmér

2018

ADNO

View full text Add to dashboard Cite

Ett problem i bedömning av skrivprov är bristande samstämmighet mellan bedömare. Att fler än en person bedömer samma skrivprov är ett vanligt sätt att öka samstämmigheten, men denna metod är tidskrävande och kostsam. Ett automatiskt bedömningsverktyg, som ger bedömaren stöd på ett effektivt och förutsägbart sätt, vore därför ett användbart hjälpmedel. I artikeln presenteras en pilotundersökning inför ett framtida arbete med automatisk bedömning av skrivprov, där användbarheten av fyra olika textmått i automatisk bedömning undersöks: textlängd, ordlängd, ordvariationsindex och nominalkvot. Materialet utgörs av två korpusar med benchmarktexter från nationella prov, Np 1 och Np 3. Varje prov ges i två olika skolämnen, svenska och svenska som andraspråk. I analysen beräknas medel- och medianvärden samt korrelationer för textmåtten och texternas betyg. Resultatet visar att skillnaderna mellan textmåttens värden i de två korpusarna är relativt stora. I Np 3 uppmäts generellt högre värden för de fyra textmåtten än i Np 1. Vidare korrelerar samtliga undersökta textmått med betyg i Np 1, medan de textmått som visar starkast korrelation i Np 1 inte på ett signifikant sätt korrelerar med betyg i Np 3. Dessutom visar analysen att texter från ett av proven, som har samma betyg men har tillkommit inom olika svenskämnen, ligger nära varandra utifrån de objektiva textmåtten. I ett framtida arbete inför automatisk bedömning av skrivprov måste användningen av textmåtten anpassas till det specifika provet. Vidare bör en automatisk bedömning omfatta många fler textegenskaper än de som mäts med de undersökta måtten.Nyckelord: skrivbedömning, automatisk bedömning, nationella prov, elevtexter, bedömning i svenska och svenska som andraspråkAnalyzing Quantity in Qualitatively Assessed Student Texts – a Future Tool for Fair Assessment?AbstractIn assessing writing one problem is the lack of rater consistency. Letting more than one rater take part in the assessment of the same test is one way of improving rater consistency, but this method is time-consuming and expensive. A tool for automated assessment, giving the human rater support in an effective and predictable way, would therefore be useful. In this article a pilot study is presented, where the usefulness of four automatic text measures in the assessment of writing tests are investigated: text length, word length, word variation and nominal ratio. The data consists of two corpora with benchmark texts from two national tests, Np 1 and Np 3. Each test is given in both Swedish and Swedish as a second language. Mean and median values are calculated, as well as correlations for the text measures and the assessment grades of the texts. The results show important differences between the values of the text measures from the two tests. In Np 3 the values for text measures are generally higher than in Np 1. Further, the four text measures correlate significantly with grades in Np 1, but the measures correlating strongest in Np 1 do not show a significant correlation in Np 3. In one of the tests, the texts with the same assessment grade but different school subjects are very similar according to the text measures. The conclusion is that a tool for automated assessment must be adapted to a specific writing test. Furthermore, an automated assessment should include the analysis of a greater amount of text qualities than those having been the focus of this study.Keywords: assessing writing, automated assessment, national tests, student texts, assessment in Swedish and Swedish as a second language

show abstract

“…Automated scoring is certainly not immune to such biases and, in fact, several studies have documented differing performance of automated scoring models for test-takers with different native languages or with disabilities (Burstein and Chodorow, 1999;Bridgeman et al, 2012;Wang and von Davier, 2014;Wang et al, 2016;Loukina and Buzick, In print).…”

Section: Detecting Biases In Automated Scoringmentioning

confidence: 99%

Building Better Open-Source Tools to Support Fairness in Automated Scoring

Madnani

Loukina

Davier³

et al. 2017

Proceedings of the First ACL Workshop on Ethics in Natural Language Processing

View full text Add to dashboard Cite

Automated scoring of written and spoken responses is an NLP application that can significantly impact lives especially when deployed as part of high-stakes tests such as the GRE® and the TOEFL®. Ethical considerations require that automated scoring algorithms treat all testtakers fairly. The educational measurement community has done significant research on fairness in assessments and automated scoring systems must incorporate their recommendations. The best way to do that is by making available automated, non-proprietary tools to NLP researchers that directly incorporate these recommendations and generate the analyses needed to help identify and resolve biases in their scoring systems. In this paper, we attempt to provide such a solution.

show abstract

Monitoring of Scoring Using the e‐rater^® Automated Scoring System and Human Raters on a Writing Test

Cited by 13 publications

References 21 publications

The many dimensions of algorithmic fairness in educational applications

The many dimensions of algorithmic fairness in educational applications

Kvantiteter i kvalitativt bedömda elevtexter – framtida verktyg för rättvis bedömning?

Building Better Open-Source Tools to Support Fairness in Automated Scoring

Contact Info

Product

Resources

About

Monitoring of Scoring Using the e‐rater® Automated Scoring System and Human Raters on a Writing Test

Cited by 13 publications

References 21 publications

The many dimensions of algorithmic fairness in educational applications

The many dimensions of algorithmic fairness in educational applications

Kvantiteter i kvalitativt bedömda elevtexter – framtida verktyg för rättvis bedömning?

Building Better Open-Source Tools to Support Fairness in Automated Scoring

Contact Info

Product

Resources

About

Monitoring of Scoring Using the e‐rater^® Automated Scoring System and Human Raters on a Writing Test