The Unreasonable Effectiveness of Transformer Language Models in Grammatical Error Correction

Abstract

Recent work on Grammatical Error Correction (GEC) has highlighted the importance of language modeling in that it is certainly possible to achieve good performance by comparing the probabilities of the proposed edits. At the same time, advancements in language modeling have managed to generate linguistic output, which is almost indistinguishable from that of human-generated text. In this paper, we up the ante by exploring the potential of more sophisticated language models in GEC and offer some key insights on their strengths and weaknesses. We show that, in line with recent results in other NLP tasks, Transformer architectures achieve consistently high performance and provide a competitive baseline for future machine learning models.

Automatic Document Sketching: Generating Drafts from Analogous Texts

“…Recent work leverages the success of large pretrained language models to generate long texts such as stories (Rashkin et al, 2020), reviews (Cho et al, 2019a and fake news (Zellers et al, 2019). Most end-user applications for assisting user writing, however, are confined to sentence-level generation Kannan et al, 2016;Alikaniotis and Raheja, 2019;Prabhumoye et al, 2019;Faltings et al, 2021). Our work focuses on document-level writing assistance in which a document sketch is constructed from a set of similar documents.…”

“…These risks have ensured that end-user applications involving text generation (e.g., Smart Compose, Smart Reply, Grammarly) still require a human to remain in control of content and are restricted to individual sentences or even smaller segments of text Figure 1: The right side shows a sketch for writing the report of a future democratic national convention, generated from a pile of previous reports. Kannan et al, 2016;Alikaniotis and Raheja, 2019;Prabhumoye et al, 2019;Faltings et al, 2021).…”

Grammatical Error Correction in Low Error Density Domains: A New Benchmark and Analyses

“…The authors use an n-gram language model, which we replace with GPT-2 (Radford et al, 2019) to see how a strong neural language model performsthis approach is similar to Alikaniotis and Raheja (2019). Hyperparameters are tuned for each dataset (see Appendix C for details).…”

Beyond Grammatical Error Correction: Improving L1-influenced research writing in English using pre-trained encoder-decoder models

“…There have been recent attempts to eliminate the time-consuming pre-training step by employing pre-trained transformer models. Alikaniotis and Raheja (2019) used pre-trained transformer models in a language-model setting, and fine-tuned BART (Lewis et al, 2020) with a small corpus of annotated sentences. Both approaches achieved results comparable to models trained with millions of sentences.…”