Ensembling and Knowledge Distilling of Large Sequence Taggers for Grammatical Error Correction

Abstract

In this paper, we investigate improvements to the GEC sequence tagging architecture with a focus on ensembling of recent cutting-edge Transformer-based encoders in Large configurations. We encourage ensembling models by majority votes on span-level edits because this approach is tolerant to the model architecture and vocabulary size. Our best ensemble achieves a new SOTA result with an F 0.5 score of 76.05 on BEA-2019 (test), even without pretraining on synthetic datasets. In addition, we perform knowledge distillation with a trained ensemble to generate new synthetic training datasets, "Troy-Blogs" and "Troy-1BW". Our best single sequence tagging model that is pretrained on the generated Troy-datasets in combination with the publicly available synthetic PIE dataset achieves a near-SOTA 1 result with an F 0.5 score of 73.21 on BEA-2019 (test). The code, datasets, and trained models are publicly available. 2 * This research was performed during Maksym Tarnavskyi's work on Ms.Sc. thesis at Ukrainian Catholic University (Tarnavskyi, 2021).1 To the best of our knowledge, our best single model gives way only to much heavier T5 model (Rothe et al., 2021).

CheckMate: English Grammatical Error Correction for Native Turkish Speakers

“…NMT approaches, which achieve state-of-the-art results, are encoder-decoder methods where encoders and decoders could be of different possible architectures such as RNNS , CNNS (Gehring et al, 2016), or Transformers (Vaswani et al, 2017), which were applied successfully on the GEC task (Yuan and Briscoe, 2016;Yuan et al, 2019;Junczys-Dowmunt et al, 2018). Recent approaches, utilize pre-trained large language models and achieve state-of-the-art results (Rothe et al, 2021;Tarnavskyi et al, 2022) by only fine-tuning them, solving the data bottleneck requirement for large networks.…”