Maximum Entropy Linear Manifold for Learning Discriminative Low-Dimensional Representation

Abstract

Representation learning is currently a very hot topic in modern machine learning, mostly due to the great success of the deep learning methods. In particular low-dimensional representation which discriminates classes can not only enhance the classification procedure, but also make it faster, while contrary to the high-dimensional embeddings can be efficiently used for visual based exploratory data analysis.In this paper we propose Maximum Entropy Linear Manifold (MELM), a multidimensional generalization of Multithreshold Entropy Linear Classifier model which is able to find a low-dimensional linear data projection maximizing discriminativeness of projected classes. As a result we obtain a linear embedding which can be used for classification, class aware dimensionality reduction and data visualization. MELM provides highly discriminative 2D projections of the data which can be used as a method for constructing robust classifiers.We provide both empirical evaluation as well as some interesting theoretical properties of our objective function such us scale and affine transformation invariance, connections with PCA and bounding of the expected balanced accuracy error.

On Loss Functions for Deep Neural Networks in Classification

“…It performs equally well or better on both MNIST and CIFAR10 in terms of both learning speed and the final performance. At the same time this information theoretic measure is very rarely used in DL community, and rather exploited in shallow learning (for both classification [3] and clustering [10]). Now we focus on the impact these losses have on noise robustness of the deep nets.…”

Forming Classifier Ensembles with Deterministic Feature Subspaces

“…Proposed method, on average, achieved slightly better results compared to its random counterpart. 3 http://scikit-learn.org/stable/ 4 https://github.com/michalkoziarski/DeterministicSubspace It is worth noting that both individual feature quality and evenness of feature distribution measures are simplified means of estimating subspace quality and diversity of ensemble, respectively. Proposing alternative, computationally feasible metrics presents possible venue of further investigation.…”

“…All experiments were implemented in Python programming language with usage of scikit-learn machine learning library 3 . In particular, all classification algorithms were taken from dedicated scikit-learn modules to ensure correctness of implementation.…”

“…The diversity itself can be ensured on several different levels. One of the most popular is to train each learner on the basis of different features, in hope that such an embedding into lower dimensions will at the same time simplify the training procedure and allow classifiers to explore different properties of supplied feature space [3]. Random Subspace (RS) [4] is the most popular implementation of this paradigm.…”