Correcting Chinese Spelling Errors with Phonetic Pre-training

Abstract

Chinese spelling correction (CSC) is an important yet challenging task. Existing state-ofthe-art methods either only use a pre-trained language model or incorporate phonological information as external knowledge. In this paper, we propose a novel end-to-end CSC model that integrates phonetic features into language model by leveraging the powerful pre-training and fine-tuning method. Instead of conventionally masking words with a special token in training language model, we replace words with phonetic features and their sound-alike words. We further propose an adaptive weighted objective to jointly train error detection and correction in a unified framework. Experimental results show that our model achieves significant improvements on SIGHAN datasets and outperforms the previous state-of-the-art methods.

Exploring and Adapting Chinese GPT to Pinyin Input Method

“…There are also task-specific BERT models, especially for the task of grammatical error correction since an important type of error is caused by characters pronounced with the same pinyin. Zhang et al (2021a) add a pinyin embedding layer and learns to predict characters from similarly pronounced candidates. PLOME add two embedding layers implemented with two GRU networks to inject both pinyin and shape of characters, respectively.…”

General and Domain Adaptive Chinese Spelling Check with Error Consistent Pretraining

“…As most spelling errors come from similar pronunciations or glyphs, Many successive studies merged the similarity knowledge into spellers. Nguyen et al [21] adopted glyph features while Cheng et al [2], Xu et al [32], Zhang et al [35] employed phonetic information. Researcher also explored the mixture method of these similarities, such as an adaptive gating module [32], GCN [2] and multi-modal [12].…”

“…Since almost all the spelling errors are related to phonological or visual similarity [16], many subsequent studies incorporated the similarity knowledge into spellers. For example, Nguyen et al [21] utilized glyph information while Cheng et al [2], Xu et al [32], Zhang et al [35] employed phonetic features. The fusion method of these similarities has also been explored, such as Graph Convolution Network (GCN) [2] and multi-modal [12,32].…”

“…To alleviate this problem, large-scale unlabelled corpus was used to enhance the spelling check ability of models. Some works [6,17,28,35] adopted multimodal-based methods to build corpora automatically for weakly supervised learning. Although some of these methods [28] work well, they still lack the consideration of the consistency between the real datasets and the automatically generated corpora, which will bring a big gap.…”

“…To build a practical competitive speller, we develop a novel general speller ECSpell which uses the Error Consistent masking strategy to exploit unlabeled data for pretraining. Previous work [17,35] ignored the significant phenomenon that the error source of a sentence could often be the only one due to the input method. The masking strategy we proposed uses this property to simulate the real scene as much as possible to automatically construct the corpus.…”