When “the state of the art” is counting words

Perelman, Les

doi:10.1016/j.asw.2014.05.001

Cited by 73 publications

(41 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, there is less controversy surrounding the use of automated systems for scoring short answer responses, as compared to essays. AES has been criticized for oversimplifying the assessment to abstract concepts that may be difficult to quantify (Condon, 2013;Perelman, 2014).…”

Section: Discussionmentioning

confidence: 99%

“…Automated scoring techniques are not without controversy, however. In particular, AES has been criticized for oversimplifying the assessment of writing to focus on rote elements, such as word count or complexity of word choice, rather than less easily quantifiable aspects, such as thoughtfulness of response or writing to a specific audience (Condon, 2013;Perelman, 2014). The application of automated scoring techniques for short answer responses is less highly contested, especially when such techniques are not used to holistically rate the quality of writing with high-stakes implications.…”

Section: Automated Scoring Techniquesmentioning

confidence: 99%

See 1 more Smart Citation

Adapting for Scalability: Automating the Video Assessment of Instructional Learning

Roberts

LoCasale-Crouch

Hamre

et al. 2017

OLJ

View full text Add to dashboard Cite

Although scalable programs, such as online courses, have the potential to reach broad audiences, they may pose challenges to evaluating learners' knowledge and skills. Automated scoring offers a possible solution. In the current paper, we describe the process of creating and testing an automated means of scoring a validated measure of teachers' observational skills, known as the Video Assessment of Instructional Learning (VAIL). Findings show that automated VAIL scores were consistently correlated with scores assigned by the hand scoring system. In addition, the automated VAIL replicated intervention effects found in the hand scoring system. The automated scoring technique appears to offer an efficient and reliable assessment. This study may offer additional insight into how to utilize similar techniques in other large-scale programs and interventions.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Automated Scoring Techniquesmentioning

confidence: 99%

Adapting for Scalability: Automating the Video Assessment of Instructional Learning

Roberts

LoCasale-Crouch

Hamre

et al. 2017

OLJ

View full text Add to dashboard Cite

show abstract

“…After an initial training for that particular prompt, some of the these otherwise highly graded responses were taken, jumbled up and then included with the training samples assigning them lowest possible grades. This was done to avoid grading those responses highly, which included a soup of gibberish keywords related to the question while not having context, connecting information (Perelman 2014).…”

Section: Augmentation Of Training Set Using Jumbled Contentmentioning

confidence: 99%

Get IT Scored Using AutoSAS — An Automated System for Scoring Short Answers

Kumar

Aggarwal

Mahata

et al. 2019

AAAI

View full text Add to dashboard Cite

In the era of MOOCs, online exams are taken by millions of candidates, where scoring short answers is an integral part. It becomes intractable to evaluate them by human graders. Thus, a generic automated system capable of grading these responses should be designed and deployed. In this paper, we present a fast, scalable, and accurate approach towards automated Short Answer Scoring (SAS). We propose and explain the design and development of a system for SAS, namely AutoSAS. Given a question along with its graded samples, AutoSAS can learn to grade that prompt successfully. This paper further lays down the features such as lexical diversity, Word2Vec, prompt, and content overlap that plays a pivotal role in building our proposed model. We also present a methodology for indicating the factors responsible for scoring an answer. The trained model is evaluated on an extensively used public dataset, namely Automated Student Assessment Prize Short Answer Scoring (ASAP-SAS). AutoSAS shows state-of-the-art performance and achieves better results by over 8% in some of the question prompts as measured by Quadratic Weighted Kappa (QWK), showing performance comparable to humans.

show abstract

“…As an example, consider that more proficient writers tend to write longer responses. Therefore, one almost always observes a consistent positive correlation between essay length and human proficiency score (Perelman, 2014;Shermis, 2014b). This is acceptable since verbal fluency -a correlate of response length -is considered an important part of the writing proficiency.…”

Section: Detecting Biases In Automated Scoringmentioning

confidence: 99%

Building Better Open-Source Tools to Support Fairness in Automated Scoring

Madnani

Loukina

Davier³

et al. 2017

Proceedings of the First ACL Workshop on Ethics in Natural Language Processing

View full text Add to dashboard Cite

Automated scoring of written and spoken responses is an NLP application that can significantly impact lives especially when deployed as part of high-stakes tests such as the GRE® and the TOEFL®. Ethical considerations require that automated scoring algorithms treat all testtakers fairly. The educational measurement community has done significant research on fairness in assessments and automated scoring systems must incorporate their recommendations. The best way to do that is by making available automated, non-proprietary tools to NLP researchers that directly incorporate these recommendations and generate the analyses needed to help identify and resolve biases in their scoring systems. In this paper, we attempt to provide such a solution.

show abstract

When “the state of the art” is counting words

Cited by 73 publications

References 5 publications

Adapting for Scalability: Automating the Video Assessment of Instructional Learning

Adapting for Scalability: Automating the Video Assessment of Instructional Learning

Get IT Scored Using AutoSAS — An Automated System for Scoring Short Answers

Building Better Open-Source Tools to Support Fairness in Automated Scoring

Contact Info

Product

Resources

About