Maximum Entropy Models with Inequality Constraints: A Case Study on Text Categorization

Abstract

Abstract. Data sparseness or overfitting is a serious problem in natural language processing employing machine learning methods. This is still true even for the maximum entropy (ME) method, whose flexible modeling capability has alleviated data sparseness more successfully than the other probabilistic models in many NLP tasks. Although we usually estimate the model so that it completely satisfies the equality constraints on feature expectations with the ME method, complete satisfaction leads to undesirable overfitting, especially for sparse features, since the constraints derived from a limited amount of training data are always uncertain. To control overfitting in ME estimation, we propose the use of box-type inequality constraints, where equality can be violated up to certain predefined levels that reflect this uncertainty. The derived models, inequality ME models, in effect have regularized estimation with L 1 norm penalties of bounded parameters. Most importantly, this regularized estimation enables the model parameters to become sparse. This can be thought of as automatic feature selection, which is expected to improve generalization performance further. We evaluate the inequality ME models on text categorization datasets, and demonstrate their advantages over standard ME estimation, similarly motivated Gaussian MAP estimation of ME models, and support vector machines (SVMs), which are one of the state-of-the-art methods for text categorization.

Multi-label maximum entropy model for social emotion classification over short text

“…The MME for social emotion classification can be extended to a generalized model with relaxing constraints and the L 2 norm penalty [13], which has also been employed in the soft-margin extension for support vector machines to improve the predictive power. The generalized model gMME is formulated as the following optimization problem:…”

“…Then, the connections between words and social emotions are estimated by the principle of maximum entropy (ME), whose flexible modeling and efficient learning capabilities have alleviated data sparseness more successfully than other probabilistic models [13]. However, the data sparseness problem can not be solved completely, even with the ME method based on all user ratings over full emotion labels (referred as multi-label maximum entropy, MME).…”

Efficient Classification with Conjunctive Features

“…(2) was set to ω/|D| (referred to as 'single width' in Ref. [22]). ); we omit the hyper-parameters when clear from contexts.…”

“…1 -regularized log-linear models ( 1 -llms) provide sparse solutions, in which weights of irrelevant features are exactly zero as a result of assuming a Laplacian prior on the weights [46], [49]. However, Kazama and Tsujii [22] have reported in a text categorization task that most features regarded as irrelevant during the training of 1 -llms appeared rarely in the task. In such a case, 1 -1 Institute of Industrial Science, the University of Tokyo a) ynaga@tkl.iis.u-tokyo.ac.jp regularization cannot greatly reduce the number of active features in each classification, while retaining the classification accuracy.…”

Text categorization based on combination of modified back propagation neural network and latent semantic analysis

“…Text categorization is a crucial and well-proven instrument to deal with and organize large numbers of textual information. In recent years, there have been extensive study and rapid progress in automatic text categorization, including Naïve Bayes [1], Decision Trees [2], K-nearest neighbor [3], maximum entropy models [4,5] and fuzzy theory based approaches [6]. Support vector machine (SVM) is very popular and proved to be one of the best algorithms for text categorization [7,8].…”