Improving Unsupervised Stemming by using Partial Lemmatization Coupled with Data-based Heuristics for Hindi

Abstract

Stemming and Lemmatization are two important natural language processing techniques widely used in Information Retrieval (IR) for query processing and in Machine Translation (MT) for reducing the data sparseness. Both minimize inflectional forms, and sometimes derivationally related forms of a word, to a common base form. Most of the existing stemmer and lemmatization work is based either on some language dependent rules which require the supervision of a language expert, or some probabilistic approach that needs vast amount of monolingual corpus, both of which develop stemming and lemmatization algorithms independently. In our work, we propose an unsupervised stemming which is hybridized with partial lemmatization for Hindi. The stemmer proposed is unique in that it exploits a novel grouping criteria & aims to improve unsupervised stemming and most importantly avoid over-stemming problem which is a usual phenomena in stemming. The later is tackled by the introduction of lemma. We incorporated lemmatization based on data heuristics obtained from the corpus, without the use of word class information. Application of this concept to unsupervised stemming yielded significant improvements in the desired results when compared to other prevailing approaches of its genre. General TermsNatural Language Processing

Improving Document Relevancy Using Integrated Language Modeling Techniques

“…From Table 1, it can be noted that all the language processing models outperformed the baseline algorithm. This is expected as the baseline algorithm returns results based on the search query only, without taking any further processing into consideration such as stemming or lemmatization [20,29].…”

“…For instance, some studies found stemming used with clustering algorithms to be beneficial in English texts [26], and also other languages [27,28]. Gupta et al [29] combined stemming with partial lemmatization for Hindi language with results indicating significant improvements than other traditional approaches. Another study compared stemming and lemmatization in clustering Finnish text documents, with results indicating the use of lemmatization to be better than stemming [30].…”

Improvisation in opinion mining using data preprocessing techniques based on consumer’s review

“…The handcrafted rule based stemming approach is easy if developer has proper linguistic knowledge on the hand the lemmatization rule can be easily produced without linguistic knowledge provided the given training data is correct [45]. Further, a comparison of lemmatization and stemming was performed in the information retrieval of documents using clustering and the result depicts that lemmatization gives best performance as compared to stemming [46].…”

Stemming and Lemmatization: A Comparison of Retrieval Performances

“…Additionally, they also found that the performance of information retrieval was better when the maximum length of lemmas is used. In 2012, Gupta et al [12] combined stemming and partial lemmatization and tested their model on the Hindi language. Their model yielded significant improvements compared to the traditional approaches.…”