Fluency Boost Learning and Inference for Neural Grammatical Error Correction

Abstract

Most of the neural sequence-to-sequence (seq2seq) models for grammatical error correction (GEC) have two limitations: (1) a seq2seq model may not be well generalized with only limited error-corrected data; (2) a seq2seq model may fail to completely correct a sentence with multiple errors through normal seq2seq inference. We attempt to address these limitations by proposing a fluency boost learning and inference mechanism. Fluency boosting learning generates fluency-boost sentence pairs during training, enabling the error correction model to learn how to improve a sentence's fluency from more instances, while fluency boosting inference allows the model to correct a sentence incrementally through multi-round seq2seq inference until the sentence's fluency stops increasing. Experiments show our approaches improve the performance of seq2seq models for GEC, achieving state-of-the-art results on both CoNLL-2014 and JFLEG benchmark datasets.

Neural Grammatical Error Correction Systems with Unsupervised Pre-training on Synthetic Data

“…Many recent advances in neural GEC aim at overcoming the mentioned data sparsity problem. Ge et al (2018a) proposed fluency-boost learning that generates additional training examples during training from an independent backward model or the forward model being trained. Xie et al (2018) sup-plied their model with noisy examples synthesized from clean sentences.…”

“…Other recent work focuses on improving model inference. Ge et al (2018a) proposed correcting a sentence more than once through multi-round model inference. Lichtarge et al (2018) introduced iterative decoding to incrementally correct a sentence with a high-precision system.…”

Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation

“…• Word2vec. Recent works on learnable evaluation metrics use simple word embeddings such as word2vec and GLoVe as input to their models (Tao et al, 2018;Lowe et al, 2017;Kannan and Vinyals, 2017). Since these static embeddings have a fixed contextindependent representation for each word, they cannot represent the rich semantics of words in contexts.…”

“…The Referenced metric and Unreferenced metric Blended Evaluation Routine (RUBER) (Tao et al, 2018) stands out from recent work in automatic dialogue evaluation, relying minimally on human-annotated datasets of response quality for training. RUBER evaluates responses with a blending of scores from two metrics: • an Unreferenced metric, which computes the relevancy of a response to a given query inspired by Grice (1975)'s theory that the quality of a response is determined by its relatedness and appropriateness, among other properties.…”

Improving Precision of Grammatical Error Correction with a Cheat Sheet

“…(Ng et al, 2013(Ng et al, , 2014. In the past few years, both GEC-tuned statistical machine translation (SMT) and neural machine translation (NMT) using sequence-to- * Equally contributed authors sequence (seq2seq) learning have demonstrated to be more effective in grammatical error correction than other approaches Ng, 2017, 2018;Ge et al, 2018;Zhao et al, 2019).…”