Feature extraction using information-theoretic learning

Abstract

A classification system typically consists of both a feature extractor (preprocessor) and a classifier. These two components can be trained either independently or simultaneously. The former option has an implementation advantage since the extractor need only be trained once for use with any classifier, whereas the latter has an advantage since it can be used to minimize classification error directly. Certain criteria, such as Minimum Classification Error, are better suited for simultaneous training, whereas other criteria, such as Mutual Information, are amenable for training the feature extractor either independently or simultaneously. Herein, an information-theoretic criterion is introduced and is evaluated for training the extractor independently of the classifier. The proposed method uses nonparametric estimation of Renyi's entropy to train the extractor by maximizing an approximation of the mutual information between the class labels and the output of the feature extractor. The evaluations show that the proposed method, even though it uses independent training, performs at least as well as three feature extraction methods that train the extractor and classifier simultaneously.

Information theoretic combination of pattern classifiers

“…Information theoretic quantities have been widely used for feature extraction and selection, e.g. Fisher et al [26], Hild et al [27] or Sindhwani et al [28], who proposed a feature selection technique for support vector machines and neural networks. Recently, Butz et al [29] proposed to apply this framework to multi-modal signal processing.…”

Variational Graph Embedding for Globally and Locally Consistent Feature Extraction

“…In contrast, our approach encodes maximization of MI as variational graph embedding, a much easier problem to solve. In addition, our approach is significantly different from the existing MI-based feature extraction algorithms such as [12,22,10], all of which are formalized as nonlinear non-convex optimization problems and do not account for the local properties of the data, as a result, cannot capture the geometric structure of the underlying manifold, which is, however, fundamental to feature learning as being demonstrated by recent researches [21,1].…”

“…which has been justified both theoretically and experimentally by many previous works, e.g., [16,22,10]. A kernel density estimator is then employed to estimate the density function involved in Eq.(3).…”

Convolutive blind source separation by minimizing mutual information between segments of signals

“…Due to the similarities of the underlying criterion, the proposed method retains the same designation, MRMI-SIG, used previously for feature extraction of static data [23] and instantaneous BSS [14]. There are four variables that must be chosen for this method.…”