TMU Transformer System Using BERT for Re-ranking at BEA 2019 Grammatical Error Correction on Restricted Track

Abstract

We introduce our system that is submitted to the restricted track of the BEA 2019 shared task on grammatical error correction 1 (GEC). It is essential to select an appropriate hypothesis sentence from the candidates list generated by the GEC model. A re-ranker can evaluate the naturalness of a corrected sentence using language models trained on large corpora. On the other hand, these language models and language representations do not explicitly take into account the grammatical errors written by learners. Thus, it is not straightforward to utilize language representations trained from a large corpus, such as Bidirectional Encoder Representations from Transformers (BERT), in a form suitable for the learner's grammatical errors. Therefore, we propose to finetune BERT on learner corpora with grammatical errors for re-ranking. The experimental results of the W&I+LOCNESS development dataset demonstrate that re-ranking using BERT can effectively improve the correction performance.

A Comprehensive Survey of Grammatical Error Correction

“…W ith the proposal of Transformer [52], many NM T based GEC models replaced traditional RNN based encoder -decoder with Transformer [53], [54], [55], [56], [57], [58], [59], [58], [60], [61]. Transformer first encodes the source sentence into a hidden state through a stack of several identical blocks, each consisting of a multi-head self-attention layer and a forward layer.…”

“…The score is calculated by summing all the n-gram log possibilities together and then normalize it by the length of hypothesis. Many systems train a 5-gram language model [50], [103], [61]. Besides, the mask ed language model probabilities computed by BERT can also be used for reranking [66].…”

“…The sentence-level correctness score could also be predicted by BERT, which is fine-tuned on learner corpus [61].…”

“…• Edit operation. Three features relating edit operation is always combined with other features, including the numbers of token-level substitutions, deletions and insertions between source and hypothesis [50], [61]. This feature could be replaced with a similar Levenshtein Distance feature [104].…”

Neural Quality Estimation Based on Multiple Hypotheses Interaction and Self-Attention for Grammatical Error Correction

“…Leshem et al [9] provided another measurement for meaning preservation using a semantic annotation scheme. Large-scale pre-trained models such as BERT [10] brings opportunities to improve the performance of GEC, demonstrating its effectiveness in context learning and enabling better quality estimation [11]. In terms of natural language generation tasks, the interaction between multiple hypotheses has an important impact on quality estimation [12], but the existing quality estimation of GEC outputs do not consider the interaction among hypotheses.…”