A Nested Attention Neural Hybrid Model for Grammatical Error Correction

Abstract

Grammatical error correction (GEC) systems strive to correct both global errors in word order and usage, and local errors in spelling and inflection. Further developing upon recent work on neural machine translation, we propose a new hybrid neural model with nested attention layers for GEC. Experiments show that the new model can effectively correct errors of both types by incorporating word and character-level information, and that the model significantly outperforms previous neural models for GEC as measured on the standard CoNLL-14 benchmark dataset. Further analysis also shows that the superiority of the proposed model can be largely attributed to the use of the nested attention mechanism, which has proven particularly effective in correcting local errors that involve small edits in orthography.

Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data

“…(Yannakoudakis et al, 2017) developed a neural sequence-labeling model for error detection to calculate the probability of each token in a sentence as being correct or incorrect, and then use the error detecting model's result as a feature to re-rank the N best hypotheses. (Ji et al, 2017) proposed a hybrid neural model incorporating both the word and character-level information. (Chollampatt and Ng, 2018) used a multilayer convolutional encoder-decoder neural network and outperforms all prior neural and statistical based systems on this task.…”

Zero-Shot Sequence Labeling: Transferring Knowledge from Sentences to Tokens

“…We use a bidirectional LSTM (Hochreiter and Schmidhuber, 1997) architecture for sentence classification, with dynamic attention over words for constructing the sentence representations. Related architectures have been successful for machine translation (Bahdanau et al, 2015), sentence summarization (Rush and Weston, 2015), entailment detection (Rocktäschel et al, 2016), and error correction (Ji et al, 2017). In this work, we modify the attention mechanism and training objective in order to make the resulting network suitable for also inferring binary token labels, while still performing well as a sentence classifier.…”

An Introductory Survey on Attention Mechanisms in NLP Problems

“…The final sentence-level representation c is then fed into a logistic regression layer to predict the category. Another type of hierarchical attention takes a top-down approach, an example of which is for grammatical error correction (Ji et al 2017). Consider a corrupted sentence: I have no enough previleges.…”