Term evaluation metrics in imbalanced text categorization

Abstract

This paper proposes four novel term evaluation metrics to represent documents in the text categorization where class distribution is imbalanced. These metrics are achieved from the revision of the four common term evaluation metrics: chi-square, information gain, odds ratio, and relevance frequency. While the common metrics require a balanced class distribution, our proposed metrics evaluate the document terms under an imbalanced distribution. They calculate the degree of relatedness of terms with respect to minor and major classes by considering their imbalanced distribution. Using these metrics in the document representation makes a better distinction between the documents of the minor and major classes and improves the performance of machine learning algorithms. The proposed metrics are assessed over three popular benchmarks (two subsets of Reuters-21578 and WebKB) by using four classification algorithms: support vector machines, naive Bayes, decision trees, and centroid-based classifiers. Our empirical results indicate that the proposed metrics outperform the common metrics in the imbalanced text categorization.

The Application of Information Classification in Agricultural Production Based on Internet of Things and Deep Learning

“…The second step is to convert the unstructured text data into computers languages. Before that, the text needs to be segmented [26]. Table 1 presents the text parts to be preprocessed and the processing measures.…”

“…The principle of the DT algorithm is to make continuous decisions according to the characteristics of the data set, and finally classify the data lines to achieve the learning effect [29], as shown in Figure 7. The Equations (24)(25)(26)(27) are used to calculate and finally get the whole DT.…”

Hybrid Leapfrog Algorithm in Feature Selection Optimization of Text Classification

“…Secondly, the requirements for objective function parameters are high. Finally, in the classification process, a large amount of sample data is needed to support its calculation and optimize performance indicators [17][18].…”