Toward Kurdish language processing: Experiments in collecting and processing the AsoSoft text corpus

Veisi, Hadi; Mohammadamini, Mohammad; Hosseini, Hawre

doi:10.1093/llc/fqy074

Cited by 16 publications

(16 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We got data from three sources to train Language Model (LM) and appreciate their word vectors. We have received more than 188 million words from the AsoSoft corpus [30] collected from various sources such as websites, textbooks, and magazines. Muhammad Azizi and AramRafeq have had a lot of difficulties collecting data on Kurdish websites1 which is about 60 million tokens.…”

Section: Central Kurdish Text Corpus For Language Modelmentioning

confidence: 99%

Central Kurdish Sentiment Analysis Using Deep Learning

Awlla¹,

Veisi²

2022

JUAPS

View full text Add to dashboard Cite

Sentiment Analysis (SA) as a type of opinion mining and as a more general topic than polarity detection, is widely used for analyzing user's reviews or comments of online expressions, which is implemented using various techniques among which the Artificial Neural Network (ANN) is the most popular one. This paper addresses the development of an SA system for the Central Kurdish language (CKB) using deep learning. Increasing the efficiency and strengthening of the SA system relies on a robust language model. In addition, for creating and training a robust language model, collecting a large amount of text corpus is required and we have created a corpus of size 300 million tokens for CKB. Also, to train the SA model, we collected 14,881 comments on Facebook, then they are labeled manually. The combination of Word2Vec for the language model and Long Short-Term Memory (LSTM) for the classifier are used to create an SA model on the CKB SA dataset. These deep learning-based techniques are the most well-known methods in this field which have received high performance in SA for various languages. The performance of the proposed method for 3 classes SA is %71.35 accuracy. This result is superior to the best-reported result for CKB..

show abstract

Section: Central Kurdish Text Corpus For Language Modelmentioning

confidence: 99%

Central Kurdish Sentiment Analysis Using Deep Learning

Awlla¹,

Veisi²

2022

JUAPS

View full text Add to dashboard Cite

show abstract

“…Collecting existing dictionaries Pronunciation dictionaries were collected from various sources, including online dictionaries for individual language varieties [17,24,25,26,27], and dictionary collections such as ipa-dict [28] and Wikipron [23]. Dictionaries from the same variety was merged.…”

Section: Multilingual Pronunciation Dictionariesmentioning

confidence: 99%

ByT5 model for massively multilingual grapheme-to-phoneme conversion

Zhu¹,

Zhang²,

Jurgens³

2022

Preprint

View full text Add to dashboard Cite

In this study, we tackle massively multilingual grapheme-tophoneme conversion through implementing G2P models based on ByT5. We have curated a G2P dataset from various sources that covers around 100 languages and trained large-scale multilingual G2P models based on ByT5. We found that ByT5 operating on byte-level inputs significantly outperformed the tokenbased mT5 model in terms of multilingual G2P. Pairwise comparison with monolingual models in these languages suggests that multilingual ByT5 models generally lower the phone error rate by jointly learning from a variety of languages. The pretrained model can further benefit low resource G2P through zero-shot prediction on unseen languages or provides pretrained weights for finetuning, which helps the model converge to a lower phone error rate than randomly initialized weights. To facilitate future research on multilingual G2P, we make available our code and pretrained multilingual G2P models at: https: //github.com/lingjzhu/CharsiuG2P.

show abstract

“…The normalization is much more important when it comes to the Kurdish Language since Kurdish writers and publishers utilize a variety of encoding schemes and orthographic standards [47] . To develop a text-to-speech system, we do text normalization, and the details of normalization completed on the text corpus are presented in [48,49] in our preprocessing stage. Some Kurdish writing includes a variety of numerical forms, such as date, time, and amounts.…”

Section: Pre-processingmentioning

confidence: 99%

End-to-End Kurdish Speech Synthesis Based on Transfer Learning

muhamad

Veisi

2022

PJBAS

Self Cite

View full text Add to dashboard Cite

A text-to-speech (TTS) system converts the texts into speech in a specific language. Several TTS systems generate natural-like speech signals in numerous languages, such as English. On the other hand, the Kurdish language has just been examined. Existing preliminary research on Kurdish speech synthesis has utilized old methods and has generated low-quality speech. They also lack important aspects of speech, including intonation, emphasis, and rhythm. Some approaches were presented to address these challenges, including the use of concatenative systems. For example, the unit selection or statistical parametric methods. On the other hand, they need a great deal of time, effort, and domain knowledge. An additional factor for Kurdish speech synthesizers' low performance is the absence of publicly available speech corpora, unlike English, which has many freely-available corpora and audiobooks. The motivation of this paper is to create a Central Kurdish speech corpus and generate a human-like speech from the Kurdish text. This paper explains how to utilize Tacotron 2, an end-to-end neural network architecture and HiFi-GAN vocoder, to produce a high-quality, realistic, and human-like Kurdish voice. This work utilizes "text, audio" pairings, which contain 10 hours of recorded audio samples and texts collected from the Internet and textbooks. It shows how to use English character embedding as the pre-trained knowledge with Kurdish characters as input and how to preprocess these audio examples to get a great outcome. Our evaluations for various types of texts show a mean opinion score of 4.1, comparable with state-of-the-art synthesizers in other languages.

show abstract

Toward Kurdish language processing: Experiments in collecting and processing the AsoSoft text corpus

Cited by 16 publications

References 3 publications

Central Kurdish Sentiment Analysis Using Deep Learning

Central Kurdish Sentiment Analysis Using Deep Learning

ByT5 model for massively multilingual grapheme-to-phoneme conversion

End-to-End Kurdish Speech Synthesis Based on Transfer Learning

Contact Info

Product

Resources

About