A new unsupervised feature selection method for text clustering based on genetic algorithms

Abstract

Nowadays a vast amount of textual information is collected and stored in various databases around the world, including the Internet as the largest database of all. This rapidly increasing growth of published text means that even the most avid reader cannot hope to keep up with all the reading in a field and consequently the nuggets of insight or new knowledge are at risk of languishing undiscovered in the literature. Text mining offers a solution to this problem by replacing or supplementing the human reader with automatic systems undeterred by the text explosion. It involves analyzing a large collection of documents to discover previously unknown information. Text clustering is one of the most important areas in text mining, which includes text preprocessing, dimension reduction by selecting some terms (features) and finally clustering using selected terms. Feature selection appears to be the most important step in the process. Conventional unsupervised feature selection methods define a measure of the discriminating power of terms to select proper terms from corpus.However up to now the valuation of terms in groups has not been investigated in reported works.In this paper a new and robust unsupervised feature selection approach is proposed that evaluates terms in groups. In addition a new Modified Term Variance measuring method is proposed for evaluating groups of terms. Furthermore a genetic based algorithm is designed and implemented for finding the most valuable groups of terms based on the new measure. These terms then will be utilized to generate the final feature vector for the clustering process . In order to evaluate and justify our approach the proposed method and also a conventional term variance method are implemented and tested using corpus collection Reuters-21578. For a more accurate comparison, methods have been tested on three corpuses and for each corpus clustering task has been done ten times and results are averaged. Results of comparing these two methods are very promising and show that our method produces better average accuracy and F1-measure than the conventional term variance method.

Dual-Tree Complex Wavelet Transform and Twin Support Vector Machine for Pathological Brain Detection

“…The reason is more features will attenuate the classification performance [51]. Reducing the number of features can simplify the model, cost shorter training time, and augment generalization performance through reduction of variance [52].…”

Advances in Document Clustering with Evolutionary-Based Algorithms

“…Nevertheless, other unsupervised feature selection methods had been used with text clustering such as Document Frequency (DF), Term Contribution (TC) or Term Variance (TV) among other statistical techniques (Luying et al, 2005;Tang et al, 2005). Moreover, there are recently evolutionary algorithm based optimization methods for term or keyword selection, such as for instance the technique in (Shamsinejadbabki and Saraee, 2012). …”

Aircraft interior failure pattern recognition utilizing text mining and neural networks

“…All the mentioned methods work in three major steps: (1) Define a formula for measuring the discriminative power of a term. (2) Sort the terms based on the value of defined measurement and (3) Choose a number of the terms from top of the list (Shamsinejadbabki and Saraee 2011).…”