Contextual Entropy and Text Categorization

Abstract

In this paper we describe a new approach to text categorization, our focus is in the amount of information (the entropy) in the text. The entropy is computed with the empirical distribution of words in the text. We provide the system with a manually segmented collection of documents in different categories. For each category a separate empirical distribution of words is computed, we will use this empirical distributions for categorization purposes.If we compute the entropy of the test document for each empirical distribution the correct category will show as a maximum. For example, if we compute the entropy of a sports document using the politics or the sports empirical word distributions then the computed entropy will be higher in sports than in politics.Our text categorization approach is simple, easy to code and needs no training time (aside from histogram computations). The classification time is linear on the size of the document and the number of document categories. We support our claims with extensive experimentation.

Fast and Accurate Language Detection in Short Texts using Contextual Entropy

“…Table 2 and figure 4 present the averaged accurate of this experiment. In figure 4 we can see the behavior of CE classifier to different values of n, that as in [4] the best results are obtaining on the first values of n.…”

“…In previous work [4] we have shown that entropy is effective in text categorization for formal as well as for intuitive reasons. The entropy is a statistic that depends both on the object itself (the text) and the context (the vocabulary distribution).…”

“…Higher order moments of the entropy has been shown to be more accurate for the task, and Contextual Entropy, defined according to equation 2 [4] optimizes for a certain value of n. This optimal classification value need to be estimated empirically.…”

“…In this work, we propose the use of an approach for text classification based on the Contextual Entropy (CE) [4], with a modification that considers that all words in each category (language) have equal probability. Thus, we compute the contextual entropy value using different word distributions, one for each document class, then classify the document as belonging to the class which maximizes the entropy (or a higher moment of it).…”

Recognition System for Home-Service-Related Sign Language Using Entropy-Based <inline-formula> <tex-math notation="LaTeX">$K$ </tex-math></inline-formula>-Means Algorithm and ABC-Based HMM

“…To calculate the entropy in (4), one needs to know the data distribution first. According to the maximum entropy criterion, if the data distribution is unknown, the possible choice is the uniform distribution [46]. This assumption may cause the estimation of the cluster number to deviate from the true number of clusters since the true data distribution is not uniform distribution.…”