Singularity and Slow Convergence of the EM algorithm for Gaussian Mixtures

Abstract

Singularities in the parameter spaces of hierarchical learning machines are known to be a main cause of slow convergence of gradient descent learning. The EM algorithm, which is another learning algorithm giving a maximum likelihood estimator, is also suffering from its slow convergence, which often appears when the component overlap is large. We analyze the dynamics of the EM algorithm for Gaussian mixtures around singularities and show that there exists a slow manifold caused by a singular structure, which is closely related to the slow convergence of the EM algorithm. We also conduct numerical simulations to confirm the theoretical analysis. Through the simulations, we compare the dynamics of the EM algorithm with that of the gradient descent algorithm, and show that their slow dynamics are caused by the same singular structure, and thus they have the same behaviors around singularities.

Model-based Clustering using Automatic Differentiation: Confronting Misspecification and High-Dimensional Data

“…In our experiments too, specifically in high-dimensional datasets, we notice that these singular points acts as Milnor attractors (Milnor, 1985;Park and Ozeki, 2009;Amari et al, 2006) -specifically, we find that naively running an AD-based inference leads to only one component with a mixing proportion close to 1 (we refer to this component as dominating component) and the remaining components with mixing proportions close to 0. More over, this dominating component changes with initialization of the parameters; which of these initialized components ends up as the dominating one depends on how close the parameters of a component are to one of the Milnor attractor points in the parameter space.…”

“…Expectation Maximization. EM avoids singularities in low-dimensions when the clusters are well separated (Park and Ozeki, 2009). In high dimensions (HD), inference is difficult as the number of parameters grows quadratically with the dimension.…”

“…Similarly, when π becomes 0 or 1, then then it is impossible to infer the parameter µ 1 & µ 2 . The Fisher information matrix becomes degenerate at these points (Amari et al, 2006;Park and Ozeki, 2009). These points are called Singularities.…”

Mining Customers' Spatio-Temporal Behavior Data Using Topographic Unsupervised Learning

“…Also very large databases can be handled by fixing T < N (this is similar as working on a random subsample of the database). This is much faster than traditional density estimator algorithms as the Kernel estimator [12] (that also needs to keep all data in memory) or the Gaussian Mixture Model [5] estimated with the EM algorithm (as the convergence speed can become extraordinarily slow [11,9]). …”

A comparison of nature inspired algorithms for multi-threshold image segmentation

“…In general, GM involves the model selection, i.e., to determine the number of components in the mixture (also called model order), and the estimation of the parameters of each component in the mixture that better adjust the statistical model. Computing the parameters of Gaussian mixtures is considered a difficult optimization task, sensitive to the initialization [37] and full of possible singularities [36]. As an optimization problem, the presented here requires an objective function, which makes use of Hellinger distance to compare the GM candidate and the original histogram.…”