Least information document representation for automated text classification

Abstract

We propose the Least Information theory (LIT) to quantify meaning of information in probability distributions and derive a new document representation model for text classification. By extending Shannon entropy to accommodate a non-linear relation between information and uncertainty, LIT offers an information-centric approach to weight terms based on probability distributions in documents vs. in the collection. We develop two term weight quantities in the document classification context: 1) LI Binary (LIB) which quantifies (least) information due to the observation of a term's (binary) occurrence in a document; and 2) LI Frequency (LIF) which measures information for the observation of a randomly picked term from the document. Both quantities are computed given term distributions in the document collection as prior knowledge and can be used separately or combined to represent documents for text classification. We conduct classification experiments on three benchmark collections, in which the proposed methods show strong performances compared to classic TF*IDF. Particularly, the LIB*LIF weighting scheme, which combines LIB and LIF, outperforms TF*IDF in several experimental settings. Despite its similarity to TF*IDF, the formulation of LIB*LIF is very different and offers a new way of thinking for modeling information processes beyond classification.

A Novel Hierarchical Document Clustering Framework on Large TREC Biomedical Documents

“…Text Rank was also developed around the same time as graph-based feature extraction. In the same vein as LexRank, TextRank is based on PageRank and uses pointless graphs, however, unlike LexRank, TextRank is limited to single documents and is a general, unsupervised extractive feature extraction system as part of the initial tokenization, the text is annotated with parts-of-speech tags, but individual words are only considered as possible additions to the graph [7]. Using this proposed method, a text unit will recommend other text units that fit within the same theme.…”

A new term‐weighting scheme for text classification using the odds of positive and negative class probabilities

“…First, two widely used data sets were used as the benchmark, as follows: The Reuters 21578 Distribution 1.0 data set (Reuters) consists of 12,902 articles and 90 topic categories from the Reuters newswire (Aphinyanaphongs et al, 2014;Debole & Sebastiani, 2003;Gliozzo et al, 2005;Ke, 2012;Sun, Lim, & Ng, 2003;Sun, Lim, & Liu, 2009;Yang & Liu, 1999;Yu et al, 2003). Following other studies by Nigam (2001) and Joachims (1998), we built binary classifiers for each class to identify the news topic.…”

“…The Newsgroups data set (NG), collected by Lang, contains about 20,000 documents that are evenly divided among 20 UseNet discussion groups (Banerjee & Basu, 2007;Gliozzo et al, 2005;Ke, 2012;McCallum & Nigam, 1998;Sun et al, 2009;Yoon, Lee, & Lee, 2006). For a fair evaluation, we evaluated our scheme using the fivefold cross-validation method.…”

When to Use Large Language Model: Upper Bound Analysis of BM25 Algorithms in Reading Comprehension Task

“…For example, Zhu et al [5,6] compiled a series of benchmarks for CTR tasks to help engineers determine whether certain methods are competitive in business. Weimao et al [7] sorted out multiple benchmarks of text classification tasks. They conducted classification experiments on this basis, so as to compare and analyze the advantages and disadvantages of various methods.…”