Stemming algorithm for Kazakh Language using rule-based approach

Sultanova, Nazerke; Kozhakhmet, Kanat; Jantayev, Ruslan; Botbayeva, Azhar

doi:10.1109/icecco48375.2019.9043253

Cited by 4 publications

(3 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is needed to have a dictionary of stemmed words. For this case, a stemming algorithm for Kazakh Language is adopted [3].…”

Section: Methodsmentioning

confidence: 99%

“…are removed, and then their stems are extracted from the remaining words. Usually, this is done using the Porter or Snowball algorithms, but for our work, we used the Kazakh rule-based stemming algorithm [3]. In the next step, the stems of the words are converted into vectors using TF-IDF method and as a result a table with numbers is formed and the tables with numbers (vectors) are built, the cosine or other measure is used to calculate the similarity of the documents.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Method for Determining the Similarity of Text Documents for the Kazakh language, Taking Into Account Synonyms: Extension to TF-IDF

Bakiyev

2022

2022 International Conference on Smart Information Systems and Technologies (SIST)

View full text Add to dashboard Cite

The task of determining the similarity of text documents has received considerable attention in many areas such as Information Retrieval, Text Mining, Natural Language Processing (NLP) and Computational Linguistics. Transferring data to numeric vectors is a complex task where algorithms such as tokenization, stopword filtering, stemming, and weighting of terms are used. The term frequency -inverse document frequency (TF-IDF) is the most widely used term weighting method to facilitate the search for relevant documents. To improve the weighting of terms, a large number of TF-IDF extensions are made. In this paper, another extension of the TF-IDF method is proposed where synonyms are taken into account. The effectiveness of the method is confirmed by experiments on functions such as Cosine, Dice and Jaccard to measure the similarity of text documents for the Kazakh language.

show abstract

“…It is needed to have a dictionary of stemmed words. For this case, a stemming algorithm for Kazakh Language is adopted [3].…”

Section: Methodsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Method for Determining the Similarity of Text Documents for the Kazakh language, Taking Into Account Synonyms: Extension to TF-IDF

Bakiyev

2022

2022 International Conference on Smart Information Systems and Technologies (SIST)

View full text Add to dashboard Cite

show abstract

“…In 2019, a stemming algorithm for the Kazakh language using the rule-based approach was provided by [15]. The stemming algorithm will be useful in the development of sentiment analysis for the Kazakh language, because of its rareness and competency.…”

Section: Literature Review and Problem Statementmentioning

confidence: 99%

Grammatical categories determination for Turkish and Kazakh languages based on machine learning algorithms and fulfilling dictionaries of link grammar parser

Yerimbetova

Tussupova²,

Sambetbayeva

et al. 2021

EEJET

View full text Add to dashboard Cite

This research is aimed at identifying the parts of speech for the Kazakh and Turkish languages in an information retrieval system. The proposed algorithms are based on machine learning techniques. In this paper, we consider the binary classification of words according to parts of speech. We decided to take the most popular machine learning algorithms. In this paper, the following approaches and well-known machine learning algorithms are studied and considered. We defined 7 dictionaries and tagged 135 million words in Kazakh and 9 dictionaries and 50 million words in the Turkish language. The main problem considered in the paper is to create algorithms for the execution of dictionaries of the so-called Link Grammar Parser (LGP) system, in particular for the Kazakh and Turkish languages, using machine learning techniques. The focus of the research is on the review and comparison of machine learning algorithms and methods that have accomplished results on various natural language processing tasks such as grammatical categories determination. For the operation of the LGP system, a dictionary is created in which a connector for each word is indicated – the type of connection that can be created using this word. The authors considered methods of filling in LGP dictionaries using machine learning. The complexities of natural language processing, however, do not exclude the possibility of identifying narrower tasks that can already be solved algorithmically: for example, determining parts of speech or splitting texts into logical groups. However, some features of natural languages significantly reduce the effectiveness of these solutions. Thus, taking into account all word forms for each word in the Kazakh and Turkish languages increases the complexity of text processing by an order of magnitude

show abstract