The Unbearable Weight of Generating Artificial Errors for Grammatical Error Correction

Htut, Phu Mon; Tetreault, Joel

doi:10.18653/v1/w19-4449

Cited by 7 publications

(3 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We discuss available datasets in Section 4.1 but it is important to note the role of synthetic data generation for GEC model training. Synthetic data has been used for GEC for a long time (Foster and Andersen, 2009;Brockett et al, 2006), and recent research shows that it can lead to significant performance gains (Stahlberg and Kumar, 2021;Htut and Tetreault, 2019). Approaches for synthetic data generation include character perturbations, dictionary-or edit-distance based replacements, shuffling word order, rule-based suffix transformations, and more (Grundkiewicz et al, 2019;Awasthi et al, 2019a;Náplava and Straka, 2019;Rothe et al, 2021b).…”

Section: Related Workmentioning

confidence: 99%

GEC-DePenD: Non-Autoregressive Grammatical Error Correction with Decoupled Permutation and Decoding

Yakovlev¹,

Podolskiy²,

Bout³

et al. 2023

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

Grammatical error correction (GEC) is an important NLP task that is currently usually solved with autoregressive sequence-tosequence models. However, approaches of this class are inherently slow due to one-byone token generation, so non-autoregressive alternatives are needed. In this work, we propose a novel non-autoregressive approach to GEC that decouples the architecture into a permutation network that outputs a self-attention weight matrix that can be used in beam search to find the best permutation of input tokens (with auxiliary ins tokens) and a decoder network based on a step-unrolled denoising autoencoder that fills in specific tokens. This allows us to find the token permutation after only one forward pass of the permutation network, avoiding autoregressive constructions. We show that the resulting network improves over previously known non-autoregressive methods for GEC and reaches the level of autoregressive methods that do not use language-specific synthetic data generation methods. Our results are supported by a comprehensive experimental validation on the ConLL-2014 and Write&Improve+LOCNESS datasets and an extensive ablation study that supports our architectural and algorithmic choices.

show abstract

Section: Related Workmentioning

confidence: 99%

GEC-DePenD: Non-Autoregressive Grammatical Error Correction with Decoupled Permutation and Decoding

Yakovlev¹,

Podolskiy²,

Bout³

et al. 2023

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

View full text Add to dashboard Cite

show abstract

“…Training GEC models is difficult due to the natural lack of suitable training data and possible erroneous corrections, so synthetic data becomes a crucial part of any GEC pipeline (Choe et al, 2019;Stahlberg and Kumar, 2021;Htut and Tetreault, 2019). It had been used for GEC even before the deep learning era that required larger datasets (Foster and Andersen, 2009;Brockett et al, 2006).…”

Section: Synthetic Datamentioning

confidence: 99%

Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule

Bout,

Podolskiy,

Nikolenko

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Progress in neural grammatical error correction (GEC) is hindered by the lack of annotated training data. Sufficient amounts of highquality manually annotated data are not available, so recent research has relied on generating synthetic data, pretraining on it, and then finetuning on real datasets; performance gains have been achieved either by ensembling or by using huge pretrained models such as XXL-T5 as the backbone. In this work, we explore an orthogonal direction: how to use available data more efficiently. First, we propose auxiliary tasks that exploit the alignment between the original and corrected sentences, such as predicting a sequence of corrections. We formulate each task as a sequence-to-sequence problem and perform multi-task training. Second, we discover that the order of datasets used for training and even individual instances within a dataset may have important effects on the final performance, so we set out to find the best training schedule. Together, these two ideas lead to significant improvements, producing results that improve state of the art with much smaller models; in particular, we outperform the best models based on T5-XXL (11B parameters) with a BART-based model (400M parameters).

show abstract

“…Experiment results showed that ungrammatical sentences generated by multi-layer CNN and Transformer were more beneficial to training GEC systems. H owever, adding too much artificial data would impact the performance [122]. So, it is important to oversample the authentic training data to maintain the balance [91].…”

Section: Back Translationmentioning

confidence: 99%

A Comprehensive Survey of Grammatical Error Correction

Wang

Dang

et al. 2021

ACM Trans. Intell. Syst. Technol.

View full text Add to dashboard Cite

Grammatical error correction (GEC) is an important application aspect of natural language processing techniques, and GEC system is a kind of very important intelligent system that has long been explored both in academic and industrial communities. The past decade has witnessed significant progress achieved in GEC for the sake of increasing popularity of machine learning and deep learning. However, there is not a survey that untangles the large amount of research works and progress in this field. We present the first survey in GEC for a comprehensive retrospective of the literature in this area. We first give the definition of GEC task and introduce the public datasets and data annotation schema. After that, we discuss six kinds of basic approaches, six commonly applied performance boosting techniques for GEC systems, and three data augmentation methods. Since GEC is typically viewed as a sister task of Machine Translation (MT), we put more emphasis on the statistical machine translation (SMT)-based approaches and neural machine translation (NMT)-based approaches for the sake of their importance. Similarly, some performance-boosting techniques are adapted from MT and are successfully combined with GEC systems for enhancement on the final performance. More importantly, after the introduction of the evaluation in GEC, we make an in-depth analysis based on empirical results in aspects of GEC approaches and GEC systems for a clearer pattern of progress in GEC, where error type analysis and system recapitulation are clearly presented. Finally, we discuss five prospective directions for future GEC researches.

show abstract

The Unbearable Weight of Generating Artificial Errors for Grammatical Error Correction

Cited by 7 publications

References 10 publications

GEC-DePenD: Non-Autoregressive Grammatical Error Correction with Decoupled Permutation and Decoding

GEC-DePenD: Non-Autoregressive Grammatical Error Correction with Decoupled Permutation and Decoding

Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule

A Comprehensive Survey of Grammatical Error Correction

Contact Info

Product

Resources

About