Automatic Assessment of English CEFR Levels Using BERT Embeddings

Schmalz, Veronica Juliana; Brutti, Alessio

doi:10.4000/books.aaccademia.10828

Cited by 4 publications

(6 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Holistic AES grading has experienced substantial advancements in recent years, driven by the integration of deep‐learning methods [21–23], which have proven effective in assessing both short and long essays, as evidenced by various studies. In the domain of short essays [24, 25], deep learning techniques have been used to achieve holistic grading, whereas for longer essays, the methods presented in [10, 11, 13] yield significant results. Additionally, [15] utilized a simple neural network model enhanced with an attention mechanism and a hierarchical structure to augment its evaluation capabilities.…”

Section: Related Workmentioning

confidence: 99%

“…Deep neural networks, such as convolutional neural networks (CNNs) [7] and recurrent neural networks [8], have been employed to learn complex patterns effectively while eliminating the need for intricate feature engineering [9][10][11]. With the advent of bidirectional encoder representations from transformers (BERT) [12], these models have been significantly enhanced in terms of AES [13]. However, most relevant studies have concentrated on generating comprehensive holistic scores to evaluate essay quality.…”

Section: Introductionmentioning

confidence: 99%

“…Furthermore, although transformer-based models, such as BERT, have shown significant potential for AES, particularly for holistic scoring [13], their capacity to perform comprehensive document-level modeling, especially for simultaneous multi-trait and holistic score predictions, remains relatively unexplored. Hence, we emphasize the critical need for an approach that not only integrates the power of transformers for holistic scoring, but also extends their capabilities to address the complexity of multi-trait evaluation.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Dual‐scale BERT using multi‐trait representations for holistic and trait‐specific essay grading

Cho,

Huang,

Kwon

2024

ETRI Journal

View full text Add to dashboard Cite

As automated essay scoring (AES) has progressed from handcrafted techniques to deep learning, holistic scoring capabilities have merged. However, specific trait assessment remains a challenge because of the limited depth of earlier methods in modeling dual assessments for holistic and multi‐trait tasks. To overcome this challenge, we explore providing comprehensive feedback while modeling the interconnections between holistic and trait representations. We introduce the DualBERT‐Trans‐CNN model, which combines transformer‐based representations with a novel dual‐scale bidirectional encoder representations from transformers (BERT) encoding approach at the document‐level. By explicitly leveraging multi‐trait representations in a multi‐task learning (MTL) framework, our DualBERT‐Trans‐CNN emphasizes the interrelation between holistic and trait‐based score predictions, aiming for improved accuracy. For validation, we conducted extensive tests on the ASAP++ and TOEFL11 datasets. Against models of the same MTL setting, ours showed a 2.0% increase in its holistic score. Additionally, compared with single‐task learning (STL) models, ours demonstrated a 3.6% enhancement in average multi‐trait performance on the ASAP++ dataset.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Dual‐scale BERT using multi‐trait representations for holistic and trait‐specific essay grading

Cho,

Huang,

Kwon

2024

ETRI Journal

View full text Add to dashboard Cite

show abstract

“…Despite its huge steps forward in NLP, GPT-3 follows trends that are already underway in AI-powered writing assistance [62,65]. With advances in language modeling, writing tools have moved toward automatic text generation.…”

Section: Automatic Text Generation and Deep Learningmentioning

confidence: 99%

AI in the Foreign Language Classroom: A Pedagogical Overview of Automated Writing Assistance Tools

Alharbi

2023

Education Research International

View full text Add to dashboard Cite

Recent technological advances in artificial intelligence (AI) have paved the way for improved and in many cases the creation of entirely new and innovative, electronic writing tools. These writing support systems assist during and after the writing process making them indispensable to many writers in general and to students in particular who can get human-like sentence completion suggestions and text generation. Although the wide adoption of these tools by students has been faced with a steady growth of scientific publications in the field, the results of these studies are often contradictory and their validity may be questioned. To gain a deeper understanding of the validity of AI-powered writing assistance tools, we conducted a systematic review of the recent empirical AI-powered writing assistance studies. The purpose of this review is twofold. First, we wanted to explore the recent scholarly publications that evaluated the use of AI-powered writing assistance tools in the classroom in terms of their types, uses, limits, and potential for improving students’ writing skills. Second, the review also sought to explore the perceptions of educators and researchers about learners’ use of AI-powered writing tools and review their recommendations on how to best ingrate these tools into the contemporary and future classroom. Using the Scopus research database, a total of 104 peer-reviewed papers were identified and analyzed. The findings indicate that students are increasingly using a variety of AI-powered writing assistance tools for improving their writing. The tools they are using can be categorized into four main groups: (1) automated writing evaluation tools, (2) tools that provide automated writing corrective feedback, (3) AI-powered machine translators, and (4) GPT-3 automatic text generators. The analysis also highlighted the scholars’ recommendations regarding dealing with learners’ use of AI-powered writing assistance tools and grouped the recommendations into two groups for researchers and educators.

show abstract

“…Sahu [59] highlighted that CBEA remains in a development stage, because it cannot properly evaluate the text structure, logic, or coherence. Results of studies on Grammarly, CYWrite, MyAccess, and Write&Improve demonstrated that human involvement for CBEA is necessary to identify errors such as disconnection between the topic and the content, since they will be attentive if the text lacks cohesion [63][64][65]. CBEA only detects programmed errors, whereas anything that is not in the program will not be detected.…”

mentioning

confidence: 99%

Synthesizing the Attributes of Computer-Based Error Analysis for ESL and EFL Learning: A Scoping Review

et al. 2022

View full text Add to dashboard Cite

An error analysis (EA) is the process of determining the incidence, nature, causes, and consequences of unsuccessful language acquisition. Traditional EA for English as a second language/English as a foreign language technique lacks an orderly investigation due to human errors. Consequently, computer-based error analysis (CBEA) was introduced into EA in linguistics to achieve accuracy and instant analysis. Although many studies have concluded that CBEA holds numerous strengths, other studies have found that CBEA has certain limitations. However, the strengths and limitations of the CBEA were not clearly synthesized and outlined. Accordingly, this review aims to explore the strengths and limitations of CBEA to provide areas for improvement of computer applications toward an efficient EA procedure. This work also aims to synthesize the strengths and limitations of CBEA mentioned in a variety of articles into a single review to sustain its efficiency and serve as a guide for teachers to benefit from the strengths and gain awareness of CBEA’s limitations. Stakeholders can access broader perspectives on developing application software capable of addressing the deficiencies in EA. By doing so, we can sustain CBEA’s efficiency for the benefit of all. For this purpose, Arksey and O’Malley’s procedure of a scoping review and the PRISMA framework were adopted to guide the filtering and selection of relevant previous studies. Sixty-two articles were selected through the processes of identification, screening, eligibility, and inclusion. Although the findings showed six strengths and seven limitations of CBEA, CBEA can only perform the diagnostic part of EA. Human intervention is still required to perform the prognostic part to accomplish an efficient EA.

show abstract

Automatic Assessment of English CEFR Levels Using BERT Embeddings

Cited by 4 publications

References 13 publications

Dual‐scale BERT using multi‐trait representations for holistic and trait‐specific essay grading

Dual‐scale BERT using multi‐trait representations for holistic and trait‐specific essay grading

AI in the Foreign Language Classroom: A Pedagogical Overview of Automated Writing Assistance Tools

Synthesizing the Attributes of Computer-Based Error Analysis for ESL and EFL Learning: A Scoping Review

Contact Info

Product

Resources

About