PLOME: Pre-training with Misspelled Knowledge for Chinese Spelling Correction

Liu, Shulin; Yang, Tao; Yue, Tianchi; Zhang, Feng; Wang, Di

doi:10.18653/v1/2021.acl-long.233

Cited by 58 publications

(59 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Works utilizing the BERT [89] encoder can utilize, or supplement, the default masking [MASK] token. The authors of [90] also used related words from confusion sets, while the authors of [91] replaced them with phonologically and visually similar ones.…”

Section: Transformer Models For Spelling Error Correctionmentioning

confidence: 99%

“…These are concatenated with word embeddings and are used in the final word encoder. For the Chinese language, [91] additionally added phonetic and shape embeddings acquired from separately-trained single-layer GRU [92] networks. Parallel to the character classification, authors also performed pronunciation prediction.…”

Section: Transformer Models For Spelling Error Correctionmentioning

confidence: 99%

See 1 more Smart Citation

Correcting Diacritics and Typos with a ByT5 Transformer Model

et al. 2022

View full text Add to dashboard Cite

Due to the fast pace of life and online communications and the prevalence of English and the QWERTY keyboard, people tend to forgo using diacritics, make typographical errors (typos) when typing in other languages. Restoring diacritics and correcting spelling is important for proper language use and the disambiguation of texts for both humans and downstream algorithms. However, both of these problems are typically addressed separately: the state-of-the-art diacritics restoration methods do not tolerate other typos, but classical spellcheckers also cannot deal adequately with all the diacritics missing.In this work, we tackle both problems at once by employing the newly-developed universal ByT5 byte-level seq2seq transformer model that requires no language-specific model structures. For a comparison, we perform diacritics restoration on benchmark datasets of 12 languages, with the addition of Lithuanian. The experimental investigation proves that our approach is able to achieve results (>98%) comparable to the previous state-of-the-art, despite being trained less and on fewer data. Our approach is also able to restore diacritics in words not seen during training with >76% accuracy. Our simultaneous diacritics restoration and typos correction approach reaches >94% alpha-word accuracy on the 13 languages. It has no direct competitors and strongly outperforms classical spell-checking or dictionary-based approaches. We also demonstrate all the accuracies to further improve with more training. Taken together, this shows the great real-world application potential of our suggested methods to more data, languages, and error classes.

show abstract

Section: Transformer Models For Spelling Error Correctionmentioning

confidence: 99%

Section: Transformer Models For Spelling Error Correctionmentioning

confidence: 99%

Correcting Diacritics and Typos with a ByT5 Transformer Model

et al. 2022

View full text Add to dashboard Cite

show abstract

“…[87] construct random rule-based generator covering the most often error categories of the Vietnamese language. Works utilizing BERT [88] encoder can utilize or supplement the default masking [MASK] token: [89] also use related words from confusion sets while [90] replace with phonologically and visually similar ones.…”

Section: Transformer Models For Spelling Error Correctionmentioning

confidence: 99%

“…These are concatenated with word embeddings and used in the final word encoder. For the Chinese language, [90] additionally add phonetic and shape embeddings, acquired from separately-trained single-layer GRU [91] networks. Parallel to character classification, authors also perform pronunciation prediction.…”

Section: Transformer Models For Spelling Error Correctionmentioning

confidence: 99%

Correcting diacritics and typos with a ByT5 transformer model

Stankevičius,

Lukoševičius,

Kapočiūtė-Dzikienė

et al. 2022

Preprint

View full text Add to dashboard Cite

Due to the fast pace of life and online communications, the prevalence of English and the QWERTY keyboard, people tend to forgo using diacritics, make typographical errors (typos) when typing. Restoring diacritics and correcting spelling is important for proper language use and disambiguation of texts for both humans and downstream algorithms. However, both of these problems are typically addressed separately, i.e., state-of-the-art diacritics restoration methods do not tolerate other typos. In this work, we tackle both problems at once by employing newly-developed ByT5 byte-level transformer models. Our simultaneous diacritics restoration and typos correction approach demonstrates near state-of-the-art performance in 13 languages, reaching >96% of the alpha-word accuracy. We also perform diacritics restoration alone on 12 benchmark datasets with the additional one for the Lithuanian language. The experimental investigation proves that our approach is able to achieve comparable results (>98%) to previously reported despite being trained on fewer data. Our approach is also able to restore diacritics in words not seen during training with >76% accuracy. We also show the accuracies to further improve with longer training. All this shows a great real-world application potential of our suggested methods to more data, languages, and error classes.

show abstract

“…To alleviate this problem, large-scale unlabelled corpus was used to enhance the spelling check ability of models. Some works [6,17,28,35] adopted multimodal-based methods to build corpora automatically for weakly supervised learning. Although some of these methods [28] work well, they still lack the consideration of the consistency between the real datasets and the automatically generated corpora, which will bring a big gap.…”

Section: Introductionmentioning

confidence: 99%

General and Domain Adaptive Chinese Spelling Check with Error Consistent Pretraining

Liu¹,

Cao²,

Geng³

et al. 2022

Preprint

View full text Add to dashboard Cite

The lack of label data is one of the significant bottlenecks for Chinese Spelling Check (CSC). Existing researches use the method of automatic generation by exploiting unlabeled data to expand the supervised corpus. However, there is a big gap between the real input scenario and automatic generated corpus. Thus, we develop a competitive general speller ECSpell which adopts the Error Consistent masking strategy to create data for pretraining. This error consistency masking strategy is used to specify the error types of automatically generated sentences which is consistent with real scene. The experimental result indicates our model outperforms previous state-of-the-art models on the general benchmark.Moreover, spellers often work within a particular domain in real life. Due to lots of uncommon domain terms, experiments on our built domain specific datasets show that general models perform terribly. Inspired by the common practice of input methods, we propose to add an alterable user dictionary to handle the zero-shot domain adaption problem. Specifically, we attach a User Dictionary guided inference module (UD) to a general token classification based speller. Our experiments demonstrate that ECSpell 𝑈 𝐷 , namely ECSpell combined with UD, surpasses all the other baselines largely, even approaching the performance on the general benchmark 1 . CCS Concepts: • Computing methodologies → Natural language processing.

show abstract

PLOME: Pre-training with Misspelled Knowledge for Chinese Spelling Correction

Cited by 58 publications

References 21 publications

Correcting Diacritics and Typos with a ByT5 Transformer Model

Correcting Diacritics and Typos with a ByT5 Transformer Model

Correcting diacritics and typos with a ByT5 transformer model

General and Domain Adaptive Chinese Spelling Check with Error Consistent Pretraining

Contact Info

Product

Resources

About