Automated Feedback and Automated Scoring in the Elementary Grades: Usage, Attitudes, and Associations with Writing Outcomes in a Districtwide Implementation of MI Write

Wilson, Joshua; Huang, Yue; Palermo, Corey; Beard, Gaysha; MacArthur, Charles A.

doi:10.1007/s40593-020-00236-w

Cited by 35 publications

(32 citation statements)

References 61 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…PEG was used solely to examine quantifiable text features of responses—no automated scores were generated. While PEG’s custom feature set comprises nearly 1,000 linguistic variables, the features initially examined for the present study were limited to the approximately 80 interpretable and instructionally relevant variables used in PEG’s application in the Automated Writing Evaluation program MI Write (see Wilson et al, 2021 ). To identify a preliminary list of features, a supervised learning approach was used to identify those variables that best explained rater accuracy (i.e., variance in agreement between the rater scores and the expert scores assigned to validity responses).…”

Section: Methodsmentioning

confidence: 99%

Rater characteristics, response content, and scoring contexts: Decomposing the determinates of scoring accuracy

Palermo¹

2022

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

Raters may introduce construct-irrelevant variance when evaluating written responses to performance assessments, threatening the validity of students’ scores. Numerous factors in the rating process, including the content of students’ responses, the characteristics of raters, and the context in which the scoring occurs, are thought to influence the quality of raters’ scores. Despite considerable study of rater effects, little research has examined the relative impacts of the factors that influence rater accuracy. In practice, such integrated examinations are needed to afford evidence-based decisions of rater selection, training, and feedback. This study provides the first naturalistic, integrated examination of rater accuracy in a large-scale assessment program. Leveraging rater monitoring data from an English language arts (ELA) summative assessment program, I specified cross-classified, multilevel models via Bayesian (i.e., Markov chain Monte Carlo) estimation to decompose the impact of response content, rater characteristics, and scoring contexts on rater accuracy. Results showed relatively little variation in accuracy attributable to teams, items, and raters. Raters did not collectively exhibit differential accuracy over time, though there was significant variation in individual rater’s scoring accuracy from response to response and day to day. I found considerable variation in accuracy across responses, which was in part explained by text features and other measures of response content that influenced scoring difficulty. Some text features differentially influenced the difficulty of scoring research and writing content. Multiple measures of raters’ qualification performance predicted their scoring accuracy, but general rater background characteristics including experience and education did not. Site-based and remote raters demonstrated comparable accuracy, while evening-shift raters were slightly less accurate, on average, than day-shift raters. This naturalistic, integrated examination of rater accuracy extends previous research and provides implications for rater recruitment, training, monitoring, and feedback to improve human evaluation of written responses.

show abstract

Section: Methodsmentioning

confidence: 99%

Rater characteristics, response content, and scoring contexts: Decomposing the determinates of scoring accuracy

Palermo¹

2022

Front. Psychol.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Recently, improvements in image and video processing have allowed for the use of AI in learning environments to track student behaviour using facial recognition, recognize gestures, and identify social interactions and emotional states (Hu, 2022). Natural language processing technologies specifically have been of interest in education, primarily for automated writing evaluation (Wilson et al, 2021). Similarly, neural networks have been applied to automate the grading of essays, including providing feedback and going beyond grammar and spelling to consider meaning and structure in the writing (Kumar & Boulanger, 2021).…”

Section: Causes Technological Driversmentioning

confidence: 99%

A Scenario for the Future of AI and Technology in Public Education

Garza¹,

Farrow²,

Lam³

et al. 2022

iJournal

View full text Add to dashboard Cite

Using narrative writing techniques, four authors use a “what if” approach to explore a future possible scenario, set in the year 2037, to reflect on looming changes in an increasingly technological human environment. The purpose of this imaginative exercise is not to pinpoint or predict future events, but to highlight large-scale forces that could push the future of learning in key directions. In this story, we follow a young student through a technology-me- diated experience of daily life. As they navigate their learning environment, we observe some of the challenges and opportunities that this student may face including personalized learning technologies, data tracking, and automated decision systems—all tools which come to firmly define this student’s envi- ronment and potential prospects. The authors follow the scenario by breaking down the major influences and forces behind this possible future reality. This analysis provides a view of the ethical dilemmas raised in such a future world and concludes with a series of proposed solutions: proactive governmental in- volvement, a reimagining of industry best practices, and an emphasis on con- text-aware, community-led, future-building initiatives.

show abstract

“…The research community has paid attention to improve the interpretability of results from CSWA without the involvement of human experts (Chapelle et al, 2015;Wei et al, 2021;Wilson et al, 2021aWilson et al, , 2021b. Interpretability of automated assessment has three major advantages.…”

Section: Introductionmentioning

confidence: 99%

Examining the Effects of a Real-Time, Knowledge-Aware Tool for Academic Writing Assessment

Ouyang

Liu

et al. 2023

Journal of Educational Computing Research

View full text Add to dashboard Cite

The computer-supported writing assessment (CSWA) has been widely used to reduce instructor workload and provide real-time feedback. Interpretability of CSWA draws extensive attention because it can benefit the validity, transparency, and knowledge-aware feedback of academic writing assessments. This study proposes a novel assessment tool, EduNERScore, which enables raters to identify different types of domain knowledge from unstructured educational texts and strengthen the interpretation of assessment through knowledge-aware evidence. This research uses an educational experiment to examine raters’ validity, efficiency, interpretability and user perception with and without the support of EduNERScore. The findings show that the experimental group with the support of EduNERScore achieves human-expert validity, high efficiency, interpretability and good use perception, compared with the control group. Finally, based on the empirical results, this research discusses the technological and educational implications for developing and applying the interpretable CSWA system in education.

show abstract

Automated Feedback and Automated Scoring in the Elementary Grades: Usage, Attitudes, and Associations with Writing Outcomes in a Districtwide Implementation of MI Write

Cited by 35 publications

References 61 publications

Rater characteristics, response content, and scoring contexts: Decomposing the determinates of scoring accuracy

Rater characteristics, response content, and scoring contexts: Decomposing the determinates of scoring accuracy

A Scenario for the Future of AI and Technology in Public Education

Examining the Effects of a Real-Time, Knowledge-Aware Tool for Academic Writing Assessment

Contact Info

Product

Resources

About