An evaluation of Reber stemmer with longest match stemmer technique in Kurdish Sorani text classification

Abstract

Stemming is one of the most significant preprocessing. stages in text categorization that most of the academic investigators aim to improve and optimize the accuracy of the classification task. High dimensionality of feature space is one of the challenges in text classification that can be decreased by many techniques. In stemming, high dimensionality of feature space is decreased by grouping those words that they have same grammatical forms and then getting their root. This work is dedicated to build an approach for Kurdish language classification using Reber Stemmer. Thus, an innovative approach is investigated to get the stem of words in Kurdish language by removing longest suffix and prefixes of words. This approach has a strong capability and meets the requirements in responding to the process of deleting as many of the required affixes as possible to get the stem of words in Kurdish language. The advantage of this stemmer is that it ignores the ordering list of affixes that receives correct stem for more than one words that have the same format. The stemming technique is implemented on KDC-4007 dataset that consists of eight classes. Support Vector Machine (SVM) and Decision Tree (DT or C 4.5) are used for the classification. This stemmer has been successfully compared with the Longest-Match stemmer technique. According to results, the F-measure of Reber stemmer and Longest-Match method in SVM is higher than DT. Reber stemmer in SVM for classes (religion, sport, health and education) obtained higher F-measure, while the rest of classes are lower in Longest-Match. Reber stemmer in DT for classes (religion, sport and art) had higher F-measure for Reber stemmer while in Longest match the rest of classes showed lower F-measure.

Medical dataset classification for Kurdish short text over social media

“…The Kurdish language is one of the languages of the Middle East that is used for speaking by Kurdish people. Central Kurdish (Sorani) and Kurmanji are two popular dialects of the Kurdish language [1 , 2] . In this project, the Sorani dialect is used for collecting the database.…”

Next word prediction based on the N-gram model for Kurdish Sorani and Kurmanji

“…Researchers have developed accurate supervised text categorization, but for the Kurdish language, text categorization is extremely difficult; challenges include complicated morphologies of the Kurdish language. The reason beyond this difficulty is the enormous utilization of inflectional and derivational affixes (Rashid et al, 2018;Saeed et al, 2018;Rashid et al, 2016). In addition, there are challenges associated with writing in the Kurdish Sorani dialect, which commonly uses suffixes and possessive pronouns (Rashid et al, 2018;Saeed et al, 2018;Rashid et al, 2016).…”

“…The reason beyond this difficulty is the enormous utilization of inflectional and derivational affixes (Rashid et al, 2018;Saeed et al, 2018;Rashid et al, 2016). In addition, there are challenges associated with writing in the Kurdish Sorani dialect, which commonly uses suffixes and possessive pronouns (Rashid et al, 2018;Saeed et al, 2018;Rashid et al, 2016). Building lemmatizers and spell checkers for Kurdish Sorani is possible because of the expansion of digital text.…”

Improving Kurdish Web Mining through Tree Data Structure and Porter’s Stemmer Algorithms

“…In this research work, KDC-4007 dataset was used to evaluate the proposed approach. SVM and decision tree (C4.5) Rashid et al, 2017;Saeed et al, 2018) are applied as two common machine learning algorithms for comparing the results. The dataset is partitioned for two inequality portions.…”