Stop Word Detection as a Binary Classification Problem

Abstract

In a wide group of languages, the stop words, which have only grammatical roles and not contributing to information content, may be simply exposed by their relatively higher occurrence frequencies. But, in agglutinative or inflectional languages, a stop word may be observed in several different surface forms due to the inflection producing noise.In this study, some of the well-known binary classification methods are employed to overcome the inflectional noise problem in stop word detection. The experiments are conducted on corpora of an agglutinative language, Turkish, in which the amount of inflection is high and a non-agglutinative language, English, in which the inflection is lower for stop words. The evaluations demonstrated that in Turkish corpus, the classification methods improve stop word detection with respect to frequency-based method. On the other hand, the classification methods applied on English corpora showed no improvement in the performance of stop word detection.

Dataset of Karakalpak language stop words

“…Both rule-based and statistical approaches to the field made it possible to automatically detect and remove stop words from given texts [5 , 9 , 12 , 15 , 16] . These methods were not limited to only English but were applied to many other languages, we concentrate on agglutinative languages in this list, such as Indonesian [16] , Turkish [4] and Bengali [17] .…”

“…There has been a lot of research already done in the task of stop word detection [2] , [3] , [4] . Most of the recent literature presents methods that are based on statistical approaches.…”

“…But, in agglutinative or inflectional languages, a stop word may be observed in several different surface forms due to the inflection producing a higher number of possible candidates, including false positives. There has been some research involving the search for stop words for agglutinative languages, such as [1 , 4 , 6] .…”

A Feature-based Approach on Automatic Stopword Detection

“…Integrating term importance to this formulation lead to use of document-based metrics such as inverse document frequency (IDF) [7]. There are also several studies that utilised a combination of multiple attributes, integrating both TF and IDF into a single metric [8,9]. Other techniques include basic voting systems that aggregate the results of a multiple sorted lists [10].…”

Dataset of stopwords extracted from Uzbek texts

“…But, in agglutinative or inflectional languages, a stopword may be observed in several different surface forms due to the inflection producing a higher number of possible candidates, including false positives. There has been some research involving the search for stopwords for agglutinative languages, such as [5 , 8] , and [7] .…”