K-MLE: A fast algorithm for learning statistical mixture models

Abstract

We describe k-MLE, a fast and efficient local search algorithm for learning finite statistical mixtures of exponential families such as Gaussian mixture models. Mixture models are traditionally learned using the expectation-maximization (EM) soft clustering technique that monotonically increases the incomplete (expected complete) likelihood. Given prescribed mixture weights, the hard clustering k-MLE algorithm iteratively assigns data to the most likely weighted component and update the component models using Maximum Likelihood Estimators (MLEs). Using the duality between exponential families and Bregman divergences, we prove that the local convergence of the complete likelihood of k-MLE follows directly from the convergence of a dual additively weighted Bregman hard clustering. The inner loop of k-MLE can be implemented using any k-means heuristic like the celebrated Lloyd's batched or Hartigan's greedy swap updates. We then show how to update the mixture weights by minimizing a crossentropy criterion that implies to update weights by taking the relative proportion of cluster points, and reiterate the mixture parameter update and mixture weight update processes until convergence. Hard EM is interpreted as a special case of k-MLE when both the component update and the weight update are performed successively in the inner loop. To initialize k-MLE, we propose k-MLE++, a careful initialization of k-MLE guaranteeing probabilistically a global bound on the best possible complete likelihood.

Collective Interaction Filtering Approach for Detection of Group in Diverse Crowded Scenes

“…will be used to seek the final group tracklet, {G}. EM [31], [32] is a frequentative method to obtain maximum probability estimates.…”

Hypothesis Testing, Information Divergence and Computational Geometry

“…This approach is limited to small dimensions. (Recently, additively-weighted Bregman Voronoi diagrams have also been used to learn mixtures of exponential families [10].) -Use non-metric tree search structures like Bregman ball trees [11] or Bregman vantage point trees [12] that can be straightforwardly extended by taking into account a weight on each point.…”

A New Implementation of k-MLE for Mixture Modeling of Wishart Distributions

“…In [1], the k-MLE algorithm is described with the Lloyd's method : Assign all observations to their closest cluster, update clusters' parameters and so on until convergence on parameters first and then on the complete likelihood. This method can produce empty clusters (especially when the number of clusters and the data dimension are large).…”

“…As remarked in [1], each component of the mixture may have its own generator F * j . It is of particular interest when the number of observations in X i is known.…”

“…penalized likelihood, cross-validation). In this paper, we adopt a greedy approach inspired by the algorithms DP-means [10] and k-MLE++ [1]. It consists in adding a cluster every time there exists an observation X i which contributes much in proportion to the kmeans F * loss function…”