Mixtures of Probabilistic Principal Component Analyzers

Abstract

Principal component analysis (PCA) is one of the most popular techniques for processing, compressing, and visualizing data, although its effectiveness is limited by its global linearity. While nonlinear variants of PCA have been proposed, an alternative paradigm is to capture data complexity by a combination of local linear PCA projections. However, conventional PCA does not correspond to a probability density, and so there is no unique way to combine PCA models. Therefore, previous attempts to formulate mixture models for PCA have been ad hoc to some extent. In this article, PCA is formulated within a maximum likelihood framework, based on a specific form of gaussian latent variable model. This leads to a well-defined mixture model for probabilistic principal component analyzers, whose parameters can be determined using an expectation-maximization algorithm. We discuss the advantages of this model in the context of clustering, density modeling, and local dimensionality reduction, and we demonstrate its application to image compression and handwritten digit recognition.

Image Denoising Using Mixtures of Projected Gaussian Scale Mixtures

“…The Mixtures of Principal Component Analyzers model from [30] and the Mixtures of Factor Analyzers model from [31] have in common with the MPGSM model that the mixtures all incorporate dimension reductions, either through PCA or Factor Analysis. However, the underlying model is different: in [30], [31] the low dimensional approximation error is considered to be Gaussian noise with a diagonal covariance matrix.…”

“…In general, q < d, such that we obtain a lower dimensional description of the observed signal vector. These models are sometimes also called generative [30], in the sense that a high-dimensional vector y j can be obtained by mapping a low-dimensional vector t j to a higher dimensional space, followed by adding a residual g j . In our application, we consider the following linear latent variable model:…”

“…Next, the complementary projection matrixV contains the d − q least dominant eigenvectors of C t and diagonal matrix Ψ consists of the d − q least dominant eigenvalues of C t . Projection onto the principal subspace, has the property that the squared reconstruction error N j=1 ||y j −ŷ j || 2 = N j=1 ||y j −Py j || 2 is minimized [30]. In the context of denoising, this allows us to only estimate the noise-free signal components in the principal subspace, followed by reconstruction.…”

“…To allow for multiple signal covariance matrices, we consider a set of k = 1, ..., K latent variable models conforming to y = V k t +V k r. Following the same reasoning as in [30], [31], we obtain mixtures as follows (called MPGSM):…”

“…in [30], [31], but for Gaussian distributed data instead of GSM distributed data, although many of the ideas presented in [30], [31] are also applicable to the MPGSM model. Compared to the MGSM model and the GSM model, the proposed MPGSM model adds a third layer of adaptation as depicted in Fig.…”