What Happens When Machines Read Our Students' Writing?

Herrington, Anne; Moran, Charles

doi:10.2307/378891

Cited by 46 publications

(18 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Scoring performance is usually measured by comparing the predicted scores on a crossvalidation set of essays to the scores assigned by human readers. For current systems, these comparisons have shown impressively high levels of agreement with human reader scores (Burstein & Chodorow, 1999;Elliot, 2002;Herrington and Moran, 2001;Landauer, Laham, & Foltz, 2002;Page, 2002), often comparable to the rates found between two human readers.…”

Section: Table Of Contentsmentioning

confidence: 78%

“…For current systems, these comparisons have shown impressively high levels of agreement with human reader scores (Burstein & Chodorow, 1999;Elliot, 2002;Herrington and Moran, 2001;Landauer, Laham, & Foltz, 2002;Page, 2002), often comparable to the rates found between two human readers. For current systems, these comparisons have shown impressively high levels of agreement with human reader scores (Burstein & Chodorow, 1999;Elliot, 2002;Herrington and Moran, 2001;Landauer, Laham, & Foltz, 2002;Page, 2002), often comparable to the rates found between two human readers.…”

Section: Introductionmentioning

confidence: 84%

See 1 more Smart Citation

Beyond Essay Length: Evaluating E‐rater®'s Performance on Toefl® Essays

Chodorow

Burstein

2004

ETS Research Report Series

View full text Add to dashboard Cite

This study examines the relation between essay length and holistic scores assigned to Test of English as a Foreign Language™ (TOEFL®) essays by e‐rater®, the automated essay scoring system developed by ETS. Results show that an early version of the system, e‐rater99, accounted for little variance in human reader scores beyond that which could be predicted by essay length. A later version of the system, e‐rater01, performs significantly better than its predecessor and is less dependent on length due to its greater reliance on measures of topical content and of complexity and diversity of vocabulary. Essay length was also examined as a possible explanation for differences in scores among examinees with native languages of Spanish, Arabic, and Japanese. Human readers and e‐rater01 show the same pattern of differences for these groups, even when effects of length are controlled.

show abstract

Section: Table Of Contentsmentioning

confidence: 78%

Section: Introductionmentioning

confidence: 84%

Beyond Essay Length: Evaluating E‐rater®'s Performance on Toefl® Essays

Chodorow

Burstein

2004

ETS Research Report Series

View full text Add to dashboard Cite

show abstract

“…However, in Huang's study (2009), 16 of the 21 participants had doubts about the AEGS they were using truly reflecting the quality of their essays. Other complaints include fixed and obscure feedback that makes learners unable to revise their essays, ease of cheating and getting high scores from AEGS by supplying words such as first, however, or in conclusion regardless the content, and word count that accounts for too large a percentage of the scoring (Herrington & Moran, 2001;Huang, 2009;Tsai, 2010). Lai (2010) compared the effectiveness of AGS with peer evaluation (peer review) and found that "EFL learners in Taiwan generally opted for peer review over AGS."…”

Section: Literature Reviewmentioning

confidence: 98%

The Consistency Between Human Raters and an Automated Essay Scoring System in Grading High School Students' English Writing

Tsai

2012

Action in Teacher Education

View full text Add to dashboard Cite

This study investigates the consistency between human raters and an automated essay scoring system in grading high school students' English compositions. A total of 923 essays from 23 classes of 12 senior high schools in Taiwan (Republic of China) were obtained and scored manually and electronically. The results show that the consistency between human raters is significantly higher than the consistency between human raters and the automated essay scoring system. To discover whether the students were confident with the automated essay grading system, a questionnaire was also distributed. The results indicate the participants realize their vocabulary level was inadequate, and they wanted to know the scores the automated essay grading system gave them along with the generated comments regarding their compositions.

show abstract

“…Burstein y Marcus (2003) explican cómo una máquina puede evaluar un criterio de buena escritura (organización) que se cree que no puede ser medido empíricamente y sostienen que los sistemas de análisis discursivo pueden identificar en forma confiable tesis y enunciados en las conclusiones. Sin embargo, Herrington y Moran (2001) sostienen que la EPC no trata a la escritura como una interacción retórica entre escritores y lectores; transmite el mensaje de que las lecturas realizadas por personas no son confiables, son irrelevantes y reemplazables; y que los rasgos superficiales de un idioma importan más que el contenido y las interacciones entre el lector y el texto. Por último, Anson (2006) se opone a la EPC diciendo que las máquinas no pueden leer discurso natural con la complejidad que los humanos lo hacen, aunque argumenta a favor de investigar cómo las tecnologías digitales pueden analizar prosa para proporcionar información útil a escritores en desarrollo.…”

Section: Informes De Los Expertosunclassified

Reflexiones y percepciones sobre la evaluación automatizada del discurso escrito

Benítez

2019

Enunciación

View full text Add to dashboard Cite

Este artículo presenta reflexiones y percepciones acerca de un potencial problema en la enseñanza y aprendizaje de la escritura en el ámbito escolar y universitario: su evaluación con instrumentos computarizados. El conflicto entre esta evaluación y la realizada por humanos puede constituir motivo de debate y análisis, como ya lo es en Estados Unidos. Allí, varios experimentos han intentado automatizar las evaluaciones de composiciones escritas. Sin embargo, los resultados han sido insatisfactorios desde el punto de vista tanto de los investigadores como de los estudiantes, a cuyos productos escritos se les ha aplicado dicha evaluación. Este artículo, por tanto, surge de la inquietud por indagar las percepciones de estudiantes universitarios acerca de su disposición a recibir retroalimentación y evaluación de sus composiciones escritas a través de un instrumento automatizado. Para recabar esta información, se les solicitó que contestaran en línea si estarían dispuestos o no a someter la evaluación de sus trabajos escritos a una computadora con la posibilidad de agregar comentarios. Las respuestas y los comentarios fueron categorizados como positivos,negativos e indecisos. Las negativas obtuvieron la mayoría, seguidas de las positivas y las indecisas. Se concluye que faltan estudios de mayor envergadura que exploren las percepciones tanto de estudiantes como de profesores de producción escrita para obtener resultados más consistentes y que las instituciones que decidan

show abstract

What Happens When Machines Read Our Students' Writing?

Cited by 46 publications

References 0 publications

Beyond Essay Length: Evaluating E‐rater®'s Performance on Toefl® Essays

Beyond Essay Length: Evaluating E‐rater®'s Performance on Toefl® Essays

The Consistency Between Human Raters and an Automated Essay Scoring System in Grading High School Students' English Writing

Reflexiones y percepciones sobre la evaluación automatizada del discurso escrito

Contact Info

Product

Resources

About