Using human judgments to examine the validity of automated grammar, syntax, and mechanical errors in writing

Crossley, Scott A.

doi:10.17239/jowr-2019.11.02.01

“…Through advances across disciplines has made it possible to conduct computational assessment of language and text comprehension that exceed those basic linguistic elements. The disciplines that advance the field of linguistic studies include computational linguistics (Graesser et al, 2014), analyses of structural and mechanical errors in texts (Crossley, Bradfield, & Bustamante, 2019), and discourse processing (Hardy & Friginal, 2016). By all means, scientific research and language technologies have been integrated, so teachers or researchers can benefit from the automated text analyses.…”

Section: An Automated Tool In Text Analysesmentioning

confidence: 99%

Utilizing an automated tool analysis to evaluate EFL students’ writing performances

Petchprasert

¹

2021

Asian. J. Second. Foreign. Lang. Educ.

View full text Add to dashboard Cite

Recently, the integration of linguistics and technology has been promoted and widely used in the field of linguistics and English writing research for several purposes. One of those purposes is to evaluate English as a Foreign Language (EFL) writing ability by using electronic assessment tools. In the current study, an automated writing evaluation tool (Coh-Metrix) was used to indicate English-major students’ writing performances based on the discourse components of the texts. The English texts generated for each writing task on two different topics were collected. The corpus analyses gathered from Coh-Metrix identified linguistic and discourse features that were interpreted to determine the 40 EFL undergraduate students’ English writing abilities. The students wrote and revised their essays in hand-written essays in class and resubmitted their essays in digital forms with corrections made. The results showed that these students demonstrated linguistic flexibility across writing assignments that they produced. The analyses also indicated that the length of the texts and the uses of the word concreteness, and the referential and deep cohesion had impacts on the students’ writing performances across the writing tasks. Besides, the findings suggest practical value in using an automated text analysis to support teachers’ instructional decisions that could help to identify improvement of students’ writing skill.

show abstract

“…Yet, the literature also puts forward that feature-based AES models may be more interpretable than deep learning ones (Amorim et al, 2018). This paper embraces the viewpoint that these two approaches can also be complementary by leveraging the state-of-the-art in NLP and automatic linguistic analysis and harnessing one of the richest pools of linguistic indices put forward in the research community (Crossley et al, 2016(Crossley et al, , 2017(Crossley et al, , 2019Kyle, 2016;Kyle et al, 2018) and applying a thorough feature selection process powered by deep learning. Moreover, the ability of deep learning of modeling complex non-linear relationships makes it particularly well-suited for AES given that the importance of a writing feature is highly dependent on its context, that is, its interactions with other writing features.…”

Section: Research Questionsmentioning

confidence: 99%

Explainable Automated Essay Scoring: Deep Learning Really Has Pedagogical Value

Kumar

¹

,

Boulanger

²

2020

View full text Add to dashboard Cite

Automated essay scoring (AES) is a compelling topic in Learning Analytics for the primary reason that recent advances in AI find it as a good testbed to explore artificial supplementation of human creativity. However, a vast swath of research tackles AES only holistically; few have even developed AES models at the rubric level, the very first layer of explanation underlying the prediction of holistic scores. Consequently, the AES black box has remained impenetrable. Although several algorithms from Explainable Artificial Intelligence have recently been published, no research has yet investigated the role that these explanation models can play in: (a) discovering the decision-making process that drives AES, (b) fine-tuning predictive models to improve generalizability and interpretability, and (c) providing personalized, formative, and fine-grained feedback to students during the writing process. Building on previous studies where models were trained to predict both the holistic and rubric scores of essays, using the Automated Student Assessment Prize's essay datasets, this study focuses on predicting the quality of the writing style of Grade-7 essays and exposes the decision processes that lead to these predictions. In doing so, it evaluates the impact of deep learning (multilayer perceptron neural networks) on the performance of AES. It has been found that the effect of deep learning can be best viewed when assessing the trustworthiness of explanation models. As more hidden layers were added to the neural network, the descriptive accuracy increased by about 10%. This study shows that faster (up to three orders of magnitude) SHAP implementations are as accurate as the slower model-agnostic one. It leverages the state-of-the-art in natural language processing, applying feature selection on a pool of 1592 linguistic indices that measure aspects of text cohesion, lexical diversity, lexical sophistication, and syntactic sophistication and complexity. In addition to the list of most globally important features, this study reports (a) a list of features that are important for a specific essay (locally), (b) a range of values for each feature that contribute to higher or lower rubric scores, and (c) a model that allows to quantify the impact of the implementation of formative feedback.

show abstract

“…Through advances across disciplines has made it possible to conduct computational assessment of language and text comprehension that exceed those basic linguistic elements. The disciplines that advance the eld of linguistic studies include computational linguistics (Graesser, et al, 2014), analyses of structural and mechanical errors in texts (Crossley, et al, 2019), and discourse processing (Hardy & Friginal, 2016). By all means, scienti c research and language technologies have been integrated, so teachers or researchers can bene t from the automated text analyses.…”

Section: An Automated Tool In Text Analysesmentioning

confidence: 99%

Utilizing an Automated Tool Analysis to Evaluate EFL Students’ Writing Performances

Petchprasert¹

2020

Preprint

View full text Add to dashboard Cite

Recently, the integration of linguistics and technology has been promoted and widely used in the field of linguistics and English writing research for several purposes. One of those purposes is to evaluate English as a Foreign Language (EFL) writing ability by using electronic assessment tools. In the current study, an automated writing evaluation tool (Coh-Metrix) was used to indicate English-major students’ writing performances based on the discourse components of the texts. The English texts generated for each writing task on two different topics were collected. The corpus analyses gathered from Coh-Metrix identified linguistic and discourse features that were interpreted to determine the 40 EFL undergraduate students’ English writing abilities. The students wrote and revised their essays in hand-written essays in class and resubmitted their essays in digital forms with corrections made. The results showed that these students demonstrated linguistic flexibility across writing assignments that they produced. The analyses also indicated that the length of the texts and the uses of the word concreteness, and the referential and deep cohesion had impacts on the students’ writing performances across the writing tasks. Besides, the findings suggest practical value in using an automated text analysis to support teachers’ instructional decisions that could help to identify improvement of students’ writing skill.

show abstract

Using human judgments to examine the validity of automated grammar, syntax, and mechanical errors in writing

Cited by 36 publications

References 19 publications

Utilizing an automated tool analysis to evaluate EFL students’ writing performances

Utilizing an automated tool analysis to evaluate EFL students’ writing performances

Explainable Automated Essay Scoring: Deep Learning Really Has Pedagogical Value

Utilizing an Automated Tool Analysis to Evaluate EFL Students’ Writing Performances

Contact Info

Product

Resources

About