A New Big Data Feature Selection Approach for Text Classification

Abstract

Feature selection (FS) is a fundamental task for text classification problems. Text feature selection aims to represent documents using the most relevant features. This process can reduce the size of datasets and improve the performance of the machine learning algorithms. Many researchers have focused on elaborating efficient FS techniques. However, most of the proposed approaches are evaluated for small datasets and validated using single machines. As textual data dimensionality becomes higher, traditional FS methods must be improved and parallelized to handle textual big data. This paper proposes a distributed approach for feature selection based on mutual information (MI) method, which is widely applied in pattern recognition and machine learning. A drawback of MI is that it ignores the frequency of the terms during the selection of features. The proposal introduces a distributed FS method, namely, Maximum Term Frequency-Mutual Information (MTF-MI), based on term frequency and mutual information techniques to improve the quality of the selected features. The proposed approach is implemented on Hadoop using the MapReduce programming model. The effectiveness of MTF-MI is demonstrated through several text classification experiments using the multinomial Naïve Bayes classifier on three datasets. Through a series of tests, the results reveal that the proposed MTF-MI method improves the classification results compared with four state-of-the-art methods in terms of macro-F1 and micro-F1 measures.

Text Classification Based on Machine Learning and Natural Language Processing Algorithms

“…For labeled data, according to the traditional support vector machine (SVM) theory, the loss function is the hinge loss, formula (10), as shown in Figure 6(a).…”

Parallel Classification Algorithm Design of Human Resource Big Data Based on Spark Platform

“…e training samples of unbalanced human resource data are smoothly updated under the state of genetic iteration, and the central point of the transmitted unbalanced data cluster is updated according to the principle of the small square function value. e obtained power spectrum density function of the transmitted unbalanced data is taken as the feature to optimize the features of the transmitted unbalanced data [15]. e process is as follows.…”

An Enhanced Ant Colony Optimization Mechanism for the Classification of Depressive Disorders

“…For an FS procedure, the entire search space encompasses all potential subsections of features, and the degree of subsections is computed using the following equation: where n denotes the number of characteristics in the existing subset of features and z is the size of the entire feature subset. FS approaches typically necessitate heuristics or randomized search tactics, which add to the complexity of the resulting group, lowering its degree of optimization problem [ 25 ]. Depending on their evaluation approach, FS techniques can be divided into three categories.…”

“…where n denotes the number of characteristics in the existing subset of features and z is the size of the entire feature subset. FS approaches typically necessitate heuristics or randomized search tactics, which add to the complexity of the resulting group, lowering its degree of optimization problem [25].…”