A fast algorithm for feature selection in conditional maximum entropy modeling

Abstract

This paper describes a fast algorithm that selects features for conditional maximum entropy modeling. Berger et al. (1996) presents an incremental feature selection (IFS) algorithm, which computes the approximate gains for all candidate features at each selection stage, and is very time-consuming for any problems with large feature spaces. In this new algorithm, instead, we only compute the approximate gains for the top-ranked features based on the models obtained from previous stages. Experiments on WSJ data in Penn Treebank are conducted to show that the new algorithm greatly speeds up the feature selection process while maintaining the same quality of selected features. One variant of this new algorithm with look-ahead functionality is also tested to further confirm the good quality of the selected features. The new algorithm is easy to implement, and given a feature space of size F, it only uses O(F) more space than the original IFS algorithm.

Maximum Entropy Models with Inequality Constraints: A Case Study on Text Categorization

“…Many other feature selection methods have been proposed both for general settings (see, e.g., Yang & Pedersen, 1997, for a comparative study of these methods for text categorization) and for ME estimation (Berger, Della Pietra, & Della Pietra, 1996;Della Pietra, Della Pietra, & Lafferty, 1997;Shirai et al, 1998;McCallum, 2003;Zhou et al, 2003). They basically order and omit (or add) features, just by observing measures for the predictive power of features such as information gain, χ 2 -test values, and gain in likelihood (Berger, Della Pietra, & Della Pietra, 1996;Della Pietra, Della Pietra, & Lafferty, 1997;McCallum, 2003;Zhou et al, 2003).…”

“…They basically order and omit (or add) features, just by observing measures for the predictive power of features such as information gain, χ 2 -test values, and gain in likelihood (Berger, Della Pietra, & Della Pietra, 1996;Della Pietra, Della Pietra, & Lafferty, 1997;McCallum, 2003;Zhou et al, 2003). The common problem with these methods is that the ordering is based on a heuristic criterion and ignores the fact that uncertainty is already contained in such measures.…”

“…Since the goal is to maximize generalization performance with the model employed or estimation algorithm, the interaction between them should be taken into account. Although the feature selection methods for ME models (Berger, Della Pietra, & Della Pietra, 1996;Della Pietra, Della Pietra, & Lafferty, 1997;Shirai et al, 1998;McCallum 2003;Zhou et al, 2003) have interaction with the estimation algorithm, they abandon giving a complete account of feature interaction by resorting to approximation to avoid the expensive calculation of measures such as gain in likelihood.…”

Conditional random fields for activity recognition

“…In both cases, the computational requirements for scoring large sets of candidate features are prohibitive. Zhou et al introduced a modification to Berger et al's algorithm to reduce the computational requirements of feature selection [118]. They note that the feature gain estimates are relatively constant between iterations.…”

“…Further work can be done to evaluate a wider collection of scoring metrics drawn from the feature selection literature. A straight forward experiment that we could try immediately would be to use the pruning method of [118], discussed in related work, to reduce the number of candidate features that we evaluate each iteration during forward selection.…”