Neural Sequence-Labelling Models for Grammatical Error Correction

Abstract

We propose an approach to N -best list reranking using neural sequence-labelling models. We train a compositional model for error detection that calculates the probability of each token in a sentence being correct or incorrect, utilising the full sentence as context. Using the error detection model, we then re-rank the N best hypotheses generated by statistical machine translation systems. Our approach achieves state-of-the-art results on error correction for three different datasets, and it has the additional advantage of only using a small set of easily computed features that require no linguistic input.

Approaching Neural Grammatical Error Correction as a Low-Resource Machine Translation Task

“…Models can also be partially initialized by pre-training monolingual language models (Ramachandran et al, 2017) or only word-embeddings (Gangi and Federico, 2017). In GEC, Yannakoudakis et al (2017) apply pretrained monolingual word-embeddings as initializations for error-detection models to re-rank SMT n-best lists. Approaches based on pre-training with monolingual data appear to be particularly wellsuited to the GEC task.…”

“…For the CoNLL 2014 benchmark on grammatical error correction , Junczys-Dowmunt and Grundkiewicz (2016) established a set of methods for GEC by SMT that remain state-of-the-art. Systems (Chollampatt and Ng, 2017;Yannakoudakis et al, 2017) that improve on results by Junczys-Dowmunt and Grundkiewicz (2016) use their set-up as a backbone for more complex systems.…”

“…Based on these developments, one would expect to see a rise of state-of-the-art neural methods for GEC, but as Junczys-Dowmunt and Grundkiewicz (2016) already noted, this is not the case. Interestingly, even today, the top systems on established GEC benchmarks are still mostly phrase-based or hybrid systems (Chollampatt and Ng, 2017;Yannakoudakis et al, 2017;Napoles and Callison-Burch, 2017). The best "pure" neural systems (Ji et al, 2017;Schmaltz et al, 2017) are several percent behind.…”

Language Model Based Grammatical Error Correction without Annotated Training Data

“…This improved the scores by about 0.3 F 0.5 in CoNLL-2014 and FCE-test Table 3: Our LM-based approach is compared against several state-of-the-art results. AMU16 SM T +LSTM and CAMB16 SM T +LSTM were both originally reported by Yannakoudakis et al (2017), while Lee and Lee (2014) is the system entered by POST in CoNLL-2014. Only our approach does not use annotated training data.…”

“…These approaches have since come to dominate the field, and a lot of recent research has focused on fine-tuning SMT systems (JunczysDowmunt and Grundkiewicz, 2016), reranking SMT output , combining SMT and classifier systems Rozovskaya and Roth, 2016), and developing various neural architectures Xie et al, 2016;Chollampatt and Ng, 2017;Yannakoudakis et al, 2017).…”