HSRA: Hindi stopword removal algorithm

Abstract

Abstract-In the last few years, electronic documents have been the main source of data in many research areas like Web Mining, Information Retrieval, Artificial Intelligence, Natural Language Processing etc.. Text Processing plays a vital role for processing structured or unstructured data from the web. Preprocessing is the main step in any text processing systems. One significant preprocessing technique is the elimination of functional words, also known as stopwords, which affects the performance of text processing tasks. An efficient stopword removal technique is required in all text processing tasks. In this paper, we are proposing a stopword removal algorithm for Hindi Language which is using the concept of a Deterministic Finite Automata (DFA). A large number of available works on stopword removal techniques are based on dictionary containing stopword lists. Then pattern matching techique is applied and the matched patterns, which is a stopword, is removed from the document. It is a time consuming task as searching process takes a long time. This makes the method inefficient and very expensive. In comparision of that, our algorithm has been tested on 200 documents and achieved 99% accuracy and also time efficient.

Construction of a generic stopwords list for Hindi language without corpus statistics

“…An approach was proposed by [11] for generating stop word list from online social network corpus in Egyptian dialect to assess the effect of removing dialect stop words on the sentiment analysis task. Another approach was presented in [12] for stop word removal from Hindi language using a deterministic finite automata. Since generally the stop words list is constructed from the statistics of a standard large-scale corpus, which is unavailable for Hindi language, therefore the issue of Hindi stop words has not been addressed yet properly.…”

Reputation Systems: Evaluating Reputation Among All Good Sellers

“…Paper [21] created a multi-domain sentiment aware dictionary using product reviews. Paper [17] is a new approach for removing stopwords from Hindi text. But the focus of these works is not on computing trust/reputation.…”

Reputation System: Evaluating Reputation among All Good Sellers

“…Review mining in Hindi Language for polarity detection is the center point for (Jha et al, 2015c), (Jha et al, 2015a), (Jha et al, 2015b), (Jha et al, 2016), (Bakliwal et al, 2012), (Narayan et al, 2002).…”