Building a Japanese Typo Dataset from Wikipedia’s Revision History

Tanaka, Yu; Murawaki, Yugo; Kawahara, Daisuke; Kurohashi, Sadao

doi:10.18653/v1/2020.acl-srw.31

Cited by 4 publications

(3 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Previous studies successfully used the Levenshtein distance to extract misspelling-correction pairs from GitHub's commit logs (Hagiwara and Mita, 2020) and Wikipedia's revision history (Tanaka et al, 2020). Although this may seem to contradict with our findings, these successes are reasonable because the text domains explored in those studies are substantially different from search query logs (Appendix E).…”

Section: Related Workcontrasting

confidence: 61%

Lattice Path Edit Distance: A Romanization-aware Edit Distance for Extracting Misspelling-Correction Pairs from Japanese Search Query Logs

Kaji

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track

View full text Add to dashboard Cite

Edit distance has been successfully used to extract training data, i.e., misspelling-correction pairs, of spelling correction models from search query logs in languages including English. However, the success does not readily apply to Japanese, where misspellings are often dissimilar to correct spellings due to the romanizationbased input methods. To address this problem, we introduce lattice path edit distance, which utilizes romanization lattices to efficiently consider all possible romanized forms of input strings. Empirical experiments using Japanese search query logs demonstrated that the lattice path edit distance outperformed baseline methods including the standard edit distance combined with an existing transliterator and morphological analyzer. A training data collection pipeline that uses the lattice path edit distance has been deployed in production at our search engine for over a year.

show abstract

Section: Related Workcontrasting

confidence: 61%

Lattice Path Edit Distance: A Romanization-aware Edit Distance for Extracting Misspelling-Correction Pairs from Japanese Search Query Logs

Kaji

2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track

View full text Add to dashboard Cite

show abstract

“…We leverage these edits to construct error-correction word dictionaries (later used to create noisy test data). Our approach to mining edits is similar to Tanaka et al (2020), but we consider multiple languages (as opposed to only Japanese), and additionally create dictionaries of word-level edits.…”

Section: Wiki-edit Miningmentioning

confidence: 99%

“…Finally for Kanji we re-use the criteria of Tanaka et al (2020) as we re-use their dataset of sentence pairs: checking if the two sentences (containing Kanji) have the same reading. Table 9 lists the number of correct/incorrect pairs (in Millions) used for noise dictionaries to create the test-sets for the various languages ( §3).…”

Section: B Chinese and Japanese Edit Miningmentioning

confidence: 99%

Robustification of Multilingual Language Models to Real-world Noise in Crosslingual Zero-shot Settings with Robust Contrastive Pretraining

Cooper Stickland,

Sengupta,

Krone

et al. 2023

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics

View full text Add to dashboard Cite

Advances in neural modeling have achieved state-of-the-art (SOTA) results on public natural language processing (NLP) benchmarks, at times surpassing human performance. However, there is a gap between public benchmarks and real-world applications where noise, such as typographical or grammatical mistakes, is abundant and can result in degraded performance. Unfortunately, works which evaluate the robustness of neural models on noisy data and propose improvements, are limited to the English language. Upon analyzing noise in different languages, we observe that noise types vary greatly across languages. Thus, existing investigations do not generalize trivially to multilingual settings. To benchmark the performance of pretrained multilingual language models, we construct noisy datasets covering five languages and four NLP tasks and observe a clear gap in the performance between clean and noisy data in the zero-shot cross-lingual setting. After investigating several ways to boost the robustness of multilingual models in this setting, we propose Robust Contrastive Pretraining (RCP). RCP combines data augmentation with a contrastive loss term at the pretraining stage and achieves large improvements on noisy (& original test data) across two sentencelevel (+3.2%) and two sequence-labeling (+10 F1-score) multilingual classification tasks.

show abstract

Sentence Pair Augmentation Approach for Grammatical Error Correction

Nagai

Maeda

2023

Lecture Notes in Electrical Engineering

View full text Add to dashboard Cite

Building a Japanese Typo Dataset from Wikipedia’s Revision History

Cited by 4 publications

References 10 publications

Lattice Path Edit Distance: A Romanization-aware Edit Distance for Extracting Misspelling-Correction Pairs from Japanese Search Query Logs

Lattice Path Edit Distance: A Romanization-aware Edit Distance for Extracting Misspelling-Correction Pairs from Japanese Search Query Logs

Robustification of Multilingual Language Models to Real-world Noise in Crosslingual Zero-shot Settings with Robust Contrastive Pretraining

Sentence Pair Augmentation Approach for Grammatical Error Correction

Contact Info

Product

Resources

About