An Adaptive EM Accelerator for Unsupervised Learning of Gaussian Mixture Models

Abstract

We propose an Anderson Acceleration (AA) scheme for the adaptive Expectation-Maximization (EM) algorithm for unsupervised learning a finite mixture model from multivariate data (Figueiredo and Jain 2002). The proposed algorithm is able to determine the optimal number of mixture components autonomously, and converges to the optimal solution much faster than its non-accelerated version. The success of the AA-based algorithm stems from several developments rather than a single breakthrough (and without these, our tests demonstrate that AA fails catastrophically). To begin, we ensure the monotonicity of the likelihood function (a the key feature of the standard EM algorithm) with a recently proposed monotonicity-control algorithm (Henderson and Varahdan 2019), enhanced by a novel monotonicity test with little overhead. We propose nimble strategies for AA to preserve the positive definiteness of the Gaussian weights and covariance matrices strictly, and to conserve up to the second moments of the observed data set exactly. Finally, we employ a K-means clustering algorithm using the gap statistic to avoid excessively overestimating the initial number of components, thereby maximizing performance. We demonstrate the accuracy and efficiency of the algorithm with several synthetic data sets that are mixtures of Gaussians distributions of known number of components, as well as data sets generated from particle-in-cell simulations. Our numerical results demonstrate speed-ups with respect to nonaccelerated EM of up to 60× when the exact number of mixture components is known, and between a few and more than an order of magnitude with component adaptivity.

Understanding and comparing risk factors and subtypes in South Korean adult and adolescent women's suicidal ideation or suicide attempt using survey and social media data

“…To understand subtypes of suicidal ideation or suicide attempt in adult and adolescent women, we used the Gaussian Mixture Model (GMM), 30 which is a probabilistic model that assumes data were generated from the finite mixture distributions. 32 To achieve interpretable results, we only considered important features obtained through a feature selection processes during building machine learning models. By doing so, we aimed to deliver more distinguishable and easy-to-interpret cluster results.…”

FedConceptEM: Robust Federated Learning Under Diverse Distribution Shifts

“…We present a generalized objective function of the Expectation-Maximization (EM) algorithm below 3 . Given M data sources {D 1 , • • • , D M }, where each local data distribution D i is a mixture of K underlying distributions, the objective function of EM (Nguyen et al, 2020) maximizes the following log-likelihood function:…”

Deep Clustering via Center-Oriented Margin Free-Triplet Loss for Skin Lesion Detection in Highly Imbalanced Datasets

“…Assuming that unknown parameters for the entire GMM are aggregated as θ, these parameters are identified by minimizing the negative log-likelihood of data samples under a positivity constraint for the mixture weights: where β is the Lagrange multiplier [51]. Once the GMM is trained, it can be used to assign each image sample to a Gaussian component:…”

“…Competing Unsupervised Methods: In the absence of label information, we demonstrated the unsupervised variant (UDC) against shallow clustering, dimensionality reduction, decomposition and deep clustering methods. We considered GMM [51] and K-Means [59] as shallow clustering baselines, principal component analysis (PCA) [60], fast independent component analysis (Fast ICA) [61] and locally linear embedding (LLE) [62] as dimensionality reduction baselines, online dictionary learning (ODL) [63] as a decomposition baseline, and a convolutional autoencoder method (CAE) [64] and traditional triplet loss as deep clustering baselines. Implementations of competing methods are described below.…”