A Nested Attention Neural Hybrid Model for Grammatical Error Correction

Ji, Jianshu; Wang, Qinlong; Toutanova, Kristina; Gong, Yongen; Truong, Steven Q. H.; Gao, Jianfeng

doi:10.48550/arxiv.1707.02026

Cited by 15 publications

(9 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In Table 4, "SMT (with LM)" refers to (Junczys-Dowmunt and Grundkiewicz, 2014) "SMT Rule-Based Hybird" refers to (Felice et al, 2014); "SMT Classification Hybird" refers to (Rozovskaya and Roth, 2016); "Neural Hybird MT" refers to (Ji et al, 2017); "CNN + EO" refers to (Chollampatt and Ng, 2018) and "EO" means rerank with edit-operation features; "Transformer + MIMs" refers to and "MIMs" means model indepent methods; "NMT SMT Hybrid" refers to ; "CNN + FB Learning" refers to (Ge et al, 2018).…”

Section: Resultsmentioning

confidence: 99%

“…(Yannakoudakis et al, 2017) developed a neural sequence-labeling model for error detection to calculate the probability of each token in a sentence as being correct or incorrect, and then use the error detecting model's result as a feature to re-rank the N best hypotheses. (Ji et al, 2017) proposed a hybrid neural model incorporating both the word and character-level information. (Chollampatt and Ng, 2018) used a multilayer convolutional encoder-decoder neural network and outperforms all prior neural and statistical based systems on this task.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data

Zhao¹,

Wang

Shen³

et al. 2019

Proceedings of the 2019 Conference of the North

171

153

View full text Add to dashboard Cite

Neural machine translation systems have become state-of-the-art approaches for Grammatical Error Correction (GEC) task. In this paper, we propose a copy-augmented architecture for the GEC task by copying the unchanged words from the source sentence to the target sentence. Since the GEC suffers from not having enough labeled training data to achieve high accuracy. We pre-train the copy-augmented architecture with a denoising auto-encoder using the unlabeled One Billion Benchmark and make comparisons between the fully pre-trained model and a partially pretrained model. It is the first time copying words from the source context and fully pretraining a sequence to sequence model are experimented on the GEC task. Moreover, We add token-level and sentence-level multi-task learning for the GEC task. The evaluation results on the CoNLL-2014 test set show that our approach outperforms all recently published state-of-the-art results by a large margin. The code and pre-trained models are released at https://github.com/zhawe01/fairseq-gec.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data

Zhao¹,

Wang

Shen³

et al. 2019

Proceedings of the 2019 Conference of the North

171

153

View full text Add to dashboard Cite

show abstract

“…Our neural classification model outperforms the CUUI system and the deep context model, and has similar performance as (Ji et al, 2017), the first best fully neural MT method. Our hybrid method consisting of our neural classification model and the public SMT system (that is, replacing the classifier method in (Rozovskaya and Roth, 2016)) has a better performance with a 50.16 F 0.5 score.…”

Section: Overall Resultsmentioning

confidence: 76%

A Simple but Effective Classification Model for Grammatical Error Correction

Kaili¹,

Wang²,

Li³

et al. 2018

Preprint

View full text Add to dashboard Cite

We treat grammatical error correction (GEC) as a classification problem in this study, where for different types of errors, a target word is identified, and the classifier predicts the correct word form from a set of possible choices. We propose a novel neural network based feature representation and classification model, trained using large text corpora without human annotations. Specifically we use RNNs with attention to represent both the left and right context of a target word. All feature embeddings are learned jointly in an end-to-end fashion. Experimental results show that our novel approach outperforms other classifier methods on the CoNLL-2014 test set (F 0.5 45.05%). Our model is simple but effective, and is suitable for industrial production.

show abstract

“…The final sentence-level representation c is then fed into a logistic regression layer to predict the category. Another type of hierarchical attention takes a top-down approach, an example of which is for grammatical error correction (Ji et al 2017). Consider a corrupted sentence: I have no enough previleges.…”

Section: (T) Imentioning

confidence: 99%

An Introductory Survey on Attention Mechanisms in NLP Problems

2019

Advances in Intelligent Systems and Computing

182

View full text Add to dashboard Cite

First derived from human intuition, later adapted to machine translation for automatic token alignment, attention mechanism, a simple method that can be used for encoding sequence data based on the importance score each element is assigned, has been widely applied to and attained significant improvement in various tasks in natural language processing, including sentiment classification, text summarization, question answering, dependency parsing, etc. In this paper, we survey through recent works and conduct an introductory summary of the attention mechanism in different NLP problems, aiming to provide our readers with basic knowledge on this widely used method, discuss its different variants for different tasks, explore its association with other techniques in machine learning, and examine methods for evaluating its performance.

show abstract

A Nested Attention Neural Hybrid Model for Grammatical Error Correction

Cited by 15 publications

References 8 publications

Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data

Improving Grammatical Error Correction via Pre-Training a Copy-Augmented Architecture with Unlabeled Data

A Simple but Effective Classification Model for Grammatical Error Correction

An Introductory Survey on Attention Mechanisms in NLP Problems

Contact Info

Product

Resources

About