TCNAEC: Advancing Sentence-Level Revision Evaluation Through Diverse Non-Native Academic English Insights

Du, Zhendong; Hashimoto, Kenji

doi:10.1109/access.2023.3342862

Cited by 2 publications

(1 citation statement)

References 22 publications

(19 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These methodologies encompass the utilization of reinforcement learning or the fine-tuning of pre-trained models to realize a SentRev system. [24] underscores that the task definition of SentRev inherently encompasses GEC, suggesting that GEC should be considered a subset of the SentRev task. This perspective highlights the integral role of grammatical accuracy within the broader scope of sentence-level revision.…”

Section: Traditional Solutionsmentioning

confidence: 99%

Exploring Sentence-level Revision Capabilities of LLMs in English for Academic Purposes Writing Assistance

Du,

Hashimoto

2024

Preprint

Self Cite

View full text Add to dashboard Cite

The English for Academic Purposes (EAP) is pivotal for scholarly communication; however, it poses significant challenges for non-native English speakers. Recently, Large Language Models (LLMs) have been extensively utilized in EAP to assist with writing tasks. EAP writing assistance typically encompasses several downstream tasks in natural language processing (NLP), such as Grammatical Error Correction (GEC). Non\-etheless, some studies have revealed that the performance of LLMs in GEC tasks is inferior to traditional GEC solutions. To explore the capabilities of LLMs more thoroughly in aspects like deep semantic and syntactic structures, this study aims to rigorously assess the performance of LLMs in the Sentence-level Revision (SentRev) task. We designed three sets of meticulous experiments to evaluate the efficacy of different LLMs. The first experiment assessed LLMs using prompts in ten different languages, finding that the SentRev performance of LLMs was heavily influenced by the language of the prompt and the quality of the input text. The second experiment investigated the performance of English LLMs with minimal prompting in the SentRev task, yet the results showed no significant changes, contradicting some prior studies. In the third experiment, we devised an innovative and straightforward method that significantly enhanced the performance of multiple LLMs by integrating academic phrases from the Formulaic Language (FL) Academic Phrasebank\footnote{\url{https://www.phrasebank.manchester.ac.uk/}}, thus overcoming the performance limitations imposed by different languages on LLMs. Additionally, our study highlights the deficiencies in existing evaluation benchmarks and suggests that higher-level, discourse-based EAP text evaluation benchmarks merit deeper exploration.

show abstract

Section: Traditional Solutionsmentioning

confidence: 99%