Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction

Abstract

This paper investigates how to effectively incorporate a pre-trained masked language model (MLM), such as BERT, into an encoderdecoder (EncDec) model for grammatical error correction (GEC). The answer to this question is not as straightforward as one might expect because the previous common methods for incorporating a MLM into an EncDec model have potential drawbacks when applied to GEC. For example, the distribution of the inputs to a GEC model can be considerably different (erroneous, clumsy, etc.) from that of the corpora used for pre-training MLMs; however, this issue is not addressed in the previous methods. Our experiments show that our proposed method, where we first fine-tune a MLM with a given GEC corpus and then use the output of the finetuned MLM as additional features in the GEC model, maximizes the benefit of the MLM. The best-performing model achieves state-ofthe-art performances on the BEA-2019 and CoNLL-2014 benchmarks. Our code is publicly available at: https://github.com/ kanekomasahiro/bert-gec.

A Self-Refinement Strategy for Noise Reduction in Grammatical Error Correction

“…Recently, several methods that incorporate pretrained masked language models such as BERT, XLNet (Yang et al, 2019), and RoBERTa into EncDec based GEC have been proposed and achieved remarkable results (Kaneko et al, 2020;Omelianchuk et al, 2020). These approaches modify the model architecture and do not directly compete with the data-driven approaches discussed in this study.…”

“…Since the reference of BEA-test is publicly unavailable, 13 See Appendix F for an ablation study of SED. 14 Improved results on the CoNLL-2014 and BEA-2019 have been appeared in arXiv less than 3 months before our submission (Kaneko et al, 2020;Omelianchuk et al, 2020) that are considered contemporaneous to our submission. More detailed experimental results, including a comparison with them, are presented in Appendix E for reference.…”

“…Comparison with existing models: a bold value denotes the best result within the column. SR and BEA indicate SR BEA+EF+L8 and BEA-test, respectively Kaneko et al (2020). andOmelianchuk et al (2020) have appeared on arXiv less than 3 months before our submission and are considered contemporaneous to our submission.…”

Improving Grammatical Error Correction with Data Augmentation by Editing Latent Representation

“…Zhao et al (2019) further applied copying mechanism (Gu et al, 2016;Jia and Liang, 2016) to Transformer. Considering the tremendous performance of pre-trained methods, pretrained language model, such as BERT (Devlin et al, 2019), can be incorporated into the encoder-decoder model (Kaneko et al, 2020).…”

Generating Diverse Corrections with Local Beam Search for Grammatical Error Correction

“…However, currently existing GEC models do not consider the generation of multiple correction candidates. Generally, in GEC, the method for obtaining multiple corrections involves the use of a plain beam search to generate the n-best candidates (Grundkiewicz et al, 2019;Kaneko et al, 2020). However, it has been shown that a plain beam search does not provide a great enough variety of candidates and produces lists of nearly identical sequences (Vijayakumar et al, 2018).…”