Comparison of the Evaluation Metrics for Neural Grammatical Error Correction With Overcorrection

Abstract

Grammar error correction (GEC) refers to the proper correction of grammatical errors in a given sentence. Important factors to consider in GEC are not only the grammatical correction of the sentence, but also the recognition of a correct sentence in which no changes are required. However, GEC approaches in which deep learning recently started being used consider only the former aspect, which leads to overcorrection, whereby changes are made to a correct sentence unnecessarily. Because this bias is also reflected in performance metrics, conventional performance metrics consider only part of the important factors in GEC. This study proposes a new metric to consider both important aspects in GEC and to provide a new viewpoint for the GEC task. To the best of the authors knowledge, this study is the first to deal with comprehensively considering the correction performance and overcorrection problem in GEC. The experimental results demonstrate that the model performance ranking was reversed when evaluating the performance with the proposed metric compared to the General Language Understanding Evaluation benchmark [21], which only considers the correction performance. This indicates that the high performance of the correction does not result in less problems with the overcorrection and that the overcorrection problem should also be considered when evaluating the model performance. Moreover, we found that the copy mechanism [14] helps to alleviate the problem of overcorrection.

Grammar System of TCFL Driven by Neural Network Technology

“…Park et al adopted the parallel structure of multiple models and used three kinds of models based on rules, statistics, and neural networks. First, they made low-level combination within categories to get category candidates and then made high-level combination of category candidates [ 20 ].…”

Design and Application of English Grammar Error Correction System Based on Deep Learning

“…Compared with the rapidly increasing number of English learners, the number of English teachers cannot increase rapidly in a short time. erefore, the use of computer-aided teaching is of great significance [1]. Automatic marking of English composition is an important aspect in CAI.…”

K-NCT: Korean Neural Grammatical Error Correction Gold-Standard Test Set Using Novel Error Type Classification Criteria

“…The reason for choosing the commercialization system for comparison is that it is a certified system used by several researchers, and the latest deep learning-based grammatical correction methodology is applied; hence, it is the most objective and reliable system for accurate analysis. The performance of each corrector is measured by using the error sentences of K-NCT as input for three commercialization systems and performing quantitative analysis using the BLEU score [20] and GLEU score [21], which are used in various deep learning-based grammatical correction studies as evaluation indicators [1,22,23]. The experimental results are shown in Table 4.…”