Self-Supervised Curriculum Learning for Spelling Error Correction

Abstract

Spelling Error Correction (SEC) that requires high-level language understanding is a challenging but useful task. Current SEC approaches normally leverage a pre-training then fine-tuning procedure that treats data equally. By contrast, Curriculum Learning (CL) utilizes training data differently during training and has shown its effectiveness in improving both performance and training efficiency in many other NLP tasks. In NMT, a model's performance has been shown sensitive to the difficulty of training examples, and CL has been shown effective to address this. In SEC, the data from different language learners are naturally distributed at different difficulty levels (some errors made by beginners are obvious to correct while some made by fluent speakers are hard), and we expect that designing a curriculum correspondingly for model learning may also help its training and bring about better performance. In this paper, we study how to further improve the performance of the state-of-the-art SEC method with CL, and propose a Self-Supervised Curriculum Learning (SSCL) approach. Specifically, we directly use the cross-entropy loss as criteria for: 1) scoring the difficulty of training data, and 2) evaluating the competence of the model. In our approach, CL improves the model training, which in return improves the CL measurement. In our experiments on the SIGHAN 2015 Chinese spelling check task, we show that SSCL is superior to previous norm-based and uncertainty-aware approaches, and establish a new state of the art (74.38% F1).

On the Effectiveness of Curriculum Learning in Educational Text Scoring

“…Depending on whether any (or both) of the two components are designed with the aid of human expertise (or data-driven approaches), a strategy can be classified as either pre-defined or automatic CL. Take difficulty measurer as an example, Platanios et al (2019) developed a pre-defined strategy in which the difficulty of input text was determined by using its length as a proxy (i.e., the longer the input text, the higher difficulty it has), while Gan, Xu, and Zan (2021) proposed an automatic strategy in which the difficulty was measured by calculating its training loss in a specific epoch.…”

“…To answer the above question, we centered our work on the design of the two key components of a CL strategy (Wang, Chen, and Zhu 2021;Liu et al 2018): (i) difficulty measurer, which determines the relative difficulty level of a training data sample; and (ii) training scheduler, which determines the data subset that should be input to a model in a specific training epoch based on the evaluation from the difficulty measurer. Inspired by previous works on proposing effective CL strategies in the broader NLP research (e.g., Spelling Error Correction (Gan, Xu, and Zan 2021) and Natural Answer Generation (Liu et al 2018)) as well as the works on automatically characterizing textual data in education, we devised two types of CL strategies in this work, i.e., pre-defined and automatic, which are grouped according to whether any or both of the two key components described above are pre-defined by human experts or automatically learned in a data-driven fashion. It should be noted that these naming terminologies are in line with those summarized by Wang, Chen, and Zhu (2021).…”

Uncovering the Risks and Drawbacks Associated With the Use of Synthetic Data for Grammatical Error Correction

“…Consequently, researchers frequently rely on synthetic data to generate these datasets. Furthermore, with the increase in the number of synthetic data studies based on self-supervised learning [20,21,22] and the demonstrated effectiveness of exponentially increasing the amount of data through scaling laws [23,24,25], there is a pressing need for in-depth validation of models trained exclusively on synthetic data.…”

K-NCT: Korean Neural Grammatical Error Correction Gold-Standard Test Set Using Novel Error Type Classification Criteria

“…From an educational point of view, interpretability is improved by adding text and examples for language learners expanding the reason for correction together [15]. Self-Supervised Curriculum Learning is applied to measure data difficulty through training loss and train the model to increase performance [16]. Applying a contrastive learning approach to the GEC model improves performance for low error density domains [17].…”