Modeling the manifolds of images of handwritten digits

Abstract

This paper describes two new methods for modeling the manifolds of digitized images of handwritten digits. The models allow a priori information about the structure of the manifolds to be combined with empirical data. Accurate modeling of the manifolds allows digits to be discriminated using the relative probability densities under the alternative models. One of the methods is grounded in principal components analysis, the other in factor analysis. Both methods are based on locally linear low-dimensional approximations to the underlying data manifold. Links with other methods that model the manifold are discussed.

Image recognition via two-dimensional random projection and nearest constrained subspace

“…If the data manifolds for all the classes can be learned, then it would be possible to design more effective classifiers. The concept of manifold has long been a powerful analytical tool for understanding image classes, for example images of human face or handwritten digits [40][41][42].…”

Learning mixture models using a genetic version of the EM algorithm

“…Learning in this framework is an estimation problem requiring an explicit probabilistic model and an algorithm for estimating the parameters of the model. An important advantage of this method is that models can be ®tted by methods like singular value decomposition (SVD) and expectationmaximization (EM) which are more e cient than gradient descent techniques (Hinton et al, 1997).…”

A unified view of probabilistic PCA and regularized linear fuzzy clustering

“…Kambhatla et al [2] proposed an iterative algorithm composed of the (hard) clustering of data sets and the estimation of local principal components in each cluster. Hinton et al [3] extended the idea to "soft version". In the "soft version", the responsibility of each data point for its generation is shared amongst all of the principal component analyzers instead of being assigned to only one analyzer.…”

“…Fukunaga et al [1] proposed local Karhunen-Loéve expansions that follows the clustering stage based on the similarities of data points. Kambhatla et al [2] and Hinton et al [3] used iterative algorithms that achieve the natural partitioning based on the reconstruction distances. And the "soft" version [3] is performed in an expectation-maximization (EM) framework [4] in which the partition assignments are considered as "missing data" and the responsibility of a principal component analyzer for each data point is estimated by using the corresponding reconstruction cost.…”

“…Kambhatla et al [2] and Hinton et al [3] used iterative algorithms that achieve the natural partitioning based on the reconstruction distances. And the "soft" version [3] is performed in an expectation-maximization (EM) framework [4] in which the partition assignments are considered as "missing data" and the responsibility of a principal component analyzer for each data point is estimated by using the corresponding reconstruction cost. Then the local models are determined by the maximization of a pseudo-likelihood function while no probability density is defined.…”