K-NCT: Korean Neural Grammatical Error Correction Gold-Standard Test Set Using Novel Error Type Classification Criteria

Abstract

Recently, active research has been conducted on Korean grammatical error correction on machine translation (MT) and automatic noise generation. However, there is no gold-standard test set for objective and official comparative analysis. A significant limitation is measuring the ill-defined performance because the experimental error types in the train set are also included in the test set. Moreover, error types in the training set are also included in the test set. Additionally, the types of errors for qualitative analysis are defined differently with no explicit guidelines. This study proposes a gold-standard test set called the Korean Neural Grammatical Correction Test set (K-NCT) for Korean grammatical error correction using a new error type classification guideline. To ensure the factuality and reliability of the proposal, we conduct a quantitative analysis using a commercialization system and human evaluation. Experimental results demonstrate that the proposed grammatical error correction test set has a well-balanced, diverse, and precise guideline. Our dataset is available at https://github.com/seonminkoo/K-NCT

Identifying MT Errors for Higher-Quality Target Language Writing

“…In response to RQ2, the findings in this study are partially identical to NMT errors provided by TAUS (2016), Koo et al (2022), andDaem et al (2017). Firstly, the lexical issues mentioned in the present study, which are essential for obtaining satisfactory translation quality, are somewhat similar to the MTPE problems provided by TAUS (2016).…”

Towards Harnessing the Most of ChatGPT for Korean Grammatical Error Correction

“…We evaluate the K-GEC capability of ChatGPT using the K-NCT dataset [21]. The K-NCT dataset consists of both erroneous and correct sentences, spanning diverse domains, phrases, and syllable counts, and encompasses varying counts and types of errors.…”

“…Therefore, Refs. [21][22][23] highlighted the shortcomings in data creation and evaluation in prior Korean grammatical correction research. They introduced an error-type classification system and, leveraging this, constructed the inaugural gold-standard test set for Korean Grammatical Error Correction (K-GEC), termed K-NCT.…”

“…Error type classification criteria for Korean grammatical and spelling error correction in K-NCT dataset[21].…”

“…Statistics of K-NCT dataset[21]. # of sents/tokens/words: number of sentences/tokens/words; △ avg of SL/WS/SS: average of sentence length/words/spaces per sentence.…”