Character-Based Machine Learning vs. Language Modeling for Diacritics Restoration

Kapočiūtė-Dzikienė, Jurgita; Davidsonas, Andrius; Vidugirienė, Aušra

doi:10.5755/j01.itc.46.4.18066

Cited by 6 publications

(7 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Linguistic studies have shown that the emergence of the current word is strongly dependent on many of the words before it [22]. Language models provide a way to calculate the probability of a string appearing.…”

Section: Semantic-based Disambiguation Model For Machine Translationmentioning

confidence: 99%

Numerical Simulation of Ambiguity Resolution in Multiple Information Streams Based on Network Machine Translation

Wang

2020

Complexity

View full text Add to dashboard Cite

In natural language, the phenomenon of polysemy is widespread, which makes it very difficult for machines to process natural language. Word sense disambiguation is a key issue in the field of natural language processing. This paper introduces the more common statistical learning methods used in the field of word sense disambiguation. Using the naive Bayesian machine learning method and the feature vector set extracted and constructed by the Dice coefficient method, a semantic word disambiguation model based on semantics is realized. The results of comparative experiments show that the proposed method is better compared with known systems. This paper proposes a method for disambiguation of word segmentation in professional fields based on unsupervised learning. This method does not rely on professional domain knowledge and training corpus and only uses the frequency, mutual information, and boundary entropy information of the string in the test corpus to solve the problem of word segmentation ambiguity. The experimental results show that these three evaluation standards can solve the problem of word segmentation ambiguity in professional fields and improve the effect of word segmentation. Among them, the segmentation result using mutual information is the best, and the performance is stable.

show abstract

Section: Semantic-based Disambiguation Model For Machine Translationmentioning

confidence: 99%

Numerical Simulation of Ambiguity Resolution in Multiple Information Streams Based on Network Machine Translation

Wang

2020

Complexity

View full text Add to dashboard Cite

show abstract

“…In [28], the character-level and word-level approaches are compared for the Lithuanian language. The authors used conditional random fields (CRF) as the sequence classifier by applying them to the character-level features.…”

Section: Character-level Approachesmentioning

confidence: 99%

Correcting Diacritics and Typos with a ByT5 Transformer Model

et al. 2022

Self Cite

View full text Add to dashboard Cite

Due to the fast pace of life and online communications and the prevalence of English and the QWERTY keyboard, people tend to forgo using diacritics, make typographical errors (typos) when typing in other languages. Restoring diacritics and correcting spelling is important for proper language use and the disambiguation of texts for both humans and downstream algorithms. However, both of these problems are typically addressed separately: the state-of-the-art diacritics restoration methods do not tolerate other typos, but classical spellcheckers also cannot deal adequately with all the diacritics missing.In this work, we tackle both problems at once by employing the newly-developed universal ByT5 byte-level seq2seq transformer model that requires no language-specific model structures. For a comparison, we perform diacritics restoration on benchmark datasets of 12 languages, with the addition of Lithuanian. The experimental investigation proves that our approach is able to achieve results (>98%) comparable to the previous state-of-the-art, despite being trained less and on fewer data. Our approach is also able to restore diacritics in words not seen during training with >76% accuracy. Our simultaneous diacritics restoration and typos correction approach reaches >94% alpha-word accuracy on the 13 languages. It has no direct competitors and strongly outperforms classical spell-checking or dictionary-based approaches. We also demonstrate all the accuracies to further improve with more training. Taken together, this shows the great real-world application potential of our suggested methods to more data, languages, and error classes.

show abstract

“…In [28] character-level and word-level approaches are compared for the Lithuanian language. The authors used conditional random fields (CRF) as the sequence classifier by applying them to the character-level features.…”

Section: Character-level Approachesmentioning

confidence: 99%

Correcting diacritics and typos with a ByT5 transformer model

Stankevičius,

Lukoševičius,

Kapočiūtė-Dzikienė

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Due to the fast pace of life and online communications, the prevalence of English and the QWERTY keyboard, people tend to forgo using diacritics, make typographical errors (typos) when typing. Restoring diacritics and correcting spelling is important for proper language use and disambiguation of texts for both humans and downstream algorithms. However, both of these problems are typically addressed separately, i.e., state-of-the-art diacritics restoration methods do not tolerate other typos. In this work, we tackle both problems at once by employing newly-developed ByT5 byte-level transformer models. Our simultaneous diacritics restoration and typos correction approach demonstrates near state-of-the-art performance in 13 languages, reaching >96% of the alpha-word accuracy. We also perform diacritics restoration alone on 12 benchmark datasets with the additional one for the Lithuanian language. The experimental investigation proves that our approach is able to achieve comparable results (>98%) to previously reported despite being trained on fewer data. Our approach is also able to restore diacritics in words not seen during training with >76% accuracy. We also show the accuracies to further improve with longer training. All this shows a great real-world application potential of our suggested methods to more data, languages, and error classes.

show abstract

Character-Based Machine Learning vs. Language Modeling for Diacritics Restoration

Cited by 6 publications

References 13 publications

Numerical Simulation of Ambiguity Resolution in Multiple Information Streams Based on Network Machine Translation

Numerical Simulation of Ambiguity Resolution in Multiple Information Streams Based on Network Machine Translation

Correcting Diacritics and Typos with a ByT5 Transformer Model

Correcting diacritics and typos with a ByT5 transformer model

Contact Info

Product

Resources

About