Clustering with Bregman Divergences

Abstract

A wide variety of distortion functions are used for clustering, e.g., squared Euclidean distance, Mahalanobis distance and relative entropy. In this paper, we propose and analyze parametric hard and soft clustering algorithms based on a large class of distortion functions known as Bregman divergences. The proposed algorithms unify centroid-based parametric clustering approaches, such as classical kmeans and information-theoretic clustering, which arise by special choices of the Bregman divergence. The algorithms maintain the simplicity and scalability of the classical kmeans algorithm, while generalizing the basic idea to a very large class of clustering loss functions. There are two main contributions in this paper. First, we pose the hard clustering problem in terms of minimizing the loss in Bregman information, a quantity motivated by rate-distortion theory, and present an algorithm to minimize this loss. Secondly, we show an explicit bijection between Bregman divergences and exponential families. The bijection enables the development of an alternative interpretation of an efficient EM scheme for learning models involving mixtures of exponential distributions. This leads to a simple soft clustering algorithm for all Bregman divergences.

The Information Geometry of Mirror Descent

“…In particular, let G : Θ → R denote a strictly convex twice-differentiable function, the divergence introduced by [7] B G : Θ × Θ → R + is: Bregman divergences are widely used in statistical inference, optimization, machine learning, and information geometry (see e.g. [2,5]). Letting Ψ(·, ·) = B G (·, ·), the mirror descent step defined is:…”

Data-driven and adaptive statistical residual evaluation for fault detection with an automotive application

“…For the particular, and present, case when the KL-divergence is used as similarity measure, an approximate solution to the clustering problem can be computed with the K-means algorithm in polynomial time [40]. For a general treatment of clustering problems with similarity measures based on Bregman divergences, including the KL-divergence, see [6].…”

Unified Real-Time Tracking and Recognition with Rotation-Invariant Fast Features

“…We use symmetric KL-divergence as the distance measure for both training and querying, since it performs better than L 2 -norm for HoG descriptors [27]. KL-divergence can be incorporated into the k-means clustering framework because it is a Bregman divergence [28]. For more robustness, we use soft-assignment of descriptors to the 3 nearest centroids, as in [25].…”