Signal Modeling and Classification Using a Robust Latent Space Model Based on $t$ Distributions

Abstract

Factor analysis is a statistical covariance modeling technique based on the assumption of normally distributed data. A mixture of factor analyzers can be hence viewed as a special case of Gaussian (normal) mixture models providing a mathematically sound framework for attribute space dimensionality reduction. A significant shortcoming of mixtures of factor analyzers is the vulnerability of normal distributions to outliers. Recently, the replacement of normal distributions with the heavier-tailed Student's-t distributions has been proposed as a way to mitigate these shortcomings and the treatment of the resulting model under an expectation-maximization (EM) algorithm framework has been conducted. In this paper we develop a Bayesian approach to factor analysis modelling based on Student'st distributions. We derive a tractable variational inference algorithm for this model by expressing the Student's-t distributed factor analyzers as a marginalization over additional latent variables. Our innovative approach provides an efficient and more robust alternative to EM-based methods, resolving their singularity and overfitting proneness problems, while allowing for the automatic determination of the optimal model size. We demonstrate the superiority of the proposed model over well-known covariance modeling techniques in a wide range of signal processing applications.

A nonparametric Bayesian approach toward robot learning by demonstration

“…Optimal model size (order) selection for finite mixture models is an important but very difficult problem which has not been completely resolved. Usually, penalized likelihood-based or entropy-based criteria are used for this purpose [20], such as the Bayesian information criterion (BIC) of Schwarz [19], and variants [22].…”

“…Hence, by maximizing 9 this lower bound L(q) (variational free energy) so that it becomes as tight as possible, not only do we minimize the KL-divergence between the true and the variational posterior, but we also implicitly integrate out the unknowns W . Due to the considered conjugate prior configuration of the DPGMR model, the variational posterior q(W ) is expected to take the same functional form as the prior, p(W ) [22]; thus, it is expected to factorize as…”

“…Derivation of the variational posterior distribution q(W ) involves maximization of the variational free energy L(q) over each one of the factors of q(W ) in turn, holding the others fixed, in an iterative manner [40]. By construction, this iterative, consecutive updating of the variational posterior distribution is guaranteed to monotonically and maximally increase the free energy L(q), which functions as the convergence criterion of the derived inference algorithm for the DPGMR model [22].…”

“…As discussed, e.g., in [22], the Student's-t distribution can be equivalently written as an infinite sum of Gaussians with the same means and scaled covariances, where the covariance scalars are Gamma-distributed latent variables:…”

“…Moreover, effectiveness of the BIC criterion is contingent on a number of conditions, which are not necessarily fulfilled in real-life application scenarios [20]; thus, BIC-based approximations are rather prone to yielding noisy model size estimates. Most significantly, likelihood-and entropy-based model selection criteria are notorious for their heavy overfitting proneness, hence often leading to over-estimation of the required model size [22].…”