Automatic Detection of Stop Words for Texts in the Uzbek Language

Abstract

Stop words are very important for information retrieval and text analysis investigation. This study aimed to automatically analyze and detect stop words in texts in the Uzbek language. Because of the limited availability of methods for automatic search of stop words of texts in Uzbek we analyzed a newly prepared corpus. The Uzbek language belongs to the family of agglutinative languages. As with all agglutinative languages, we can explain that the detection of stop words in Uzbek texts is a more complex process than in inflected languages: In inflected languages, words such as auxiliary words, articles, prepositions can be included in the stop words group. In agglutinative languages, the meanings of such words are hidden in the text. Therefore, it is not appropriate to apply all known methods of stop words detection in inflected languages directly to agglutinative languages. In this work, the “School corpus” which contains 731156 Uzbek words has been investigated. The bigram method of analysis was applied to the corpus. We proposed the collocation method of detecting stop words of the corpus. We proposed the method of automatically detecting stop words of texts in Uzbek. It is shown that the collocation method is 6 times better than the bigram method.

A machine transliteration tool between Uzbek alphabets

“…Transliteration between Cyrillic and Latin alphabets of Uzbek language has been done by Mansurovs [9], who used a data-driven approach, by aligning words and training a decision-tree classifier. Among some other NLP work that has been done on low-resource Uzbek language so far, there are a morphological analyzer [10], WordNet type synsets [11], Uzbek stopwords dataset [12], sentiment analysis and text classification [13,14,15], cross-lingual word-embeddings [16], as well as a pretrained Uzbek language model based on the BERT architecture [17].…”

Uzbek Sentiment Analysis based on local Restaurant Reviews

“…Then, we applied stop words to remove low-level information words from our comments to focus on important information. The technique is based on [8] paper where it is a proposed algorithm of automatic detection of single word stop words collection using TFIDF(Term frequencyinverse document frequency). After that, each word is processed to lexicon-free stemming tool [7] algorithm for decreasing the word capacity because of prefixes and suffixes.…”

“…The first steps are removing URLs, punctuation, and lower-casing. The second step is ignoring stopwords [8] from the dataset where it is based on accuracy evaluation after generating the list of stop words using the TF-IDF algorithm; Then, we applied the stemming algorithm [7,9] which is based on Uzbek words' endings' electronic dictionary that uses combinatorial approach inferring apply for part of speech of the Uzbek language: nouns, adjectives, numerals, verbs, participles, moods, voices. Advantages of using the algorithm are lexicon-free and its complexity that allows one operation (referring to the dictionary of endings of the language) to perform: segmentation of the word into suffixes; performs morphological analysis of the word.…”