On the uniform concentration bounds and large sample properties of clustering with Bregman divergences

Abstract

Clustering with Bregman divergence has been used in literature to unify centroid‐based parametric clustering approaches and to allow the detection of nonspherical clusters within the data. Although empirically useful, the large sample theoretical aspects of Bregman clustering techniques remain largely unexplored. In this paper, we attempt to bridge the gap between the theory and practice of centroid‐based Bregman hard clustering by providing uniform deviation bounds on the clustering objective. Our theoretical analysis relies on the famous Vapnik–Chervonenkis (VC) theory, which, although has been extensively used in a supervised learning context, remains largely unexplored in finding empirical risks for unsupervised learning scenarios. As opposed to most of the theoretical works on clustering, our framework allows the number of features (p) to vary with the number of observations (n). The strong consistency of the sample cluster centroids, under standard assumptions in the literature, also follows as a corollary under this general framework. Furthermore, we show that the rate of convergence is at most of the order scriptO)(normallognn, under the standard regularity conditions.

Uniform Concentration Bounds toward a Unified Framework for Robust Clustering

“…Theorem 3.3 states that indeed strong consistency holds, with convergence rate O(n −1/2 ). Note that this rate is faster than that found previously by Paul et al (2021a). Before we proceed, recall we say that X n = O P (a n ) if the sequence X n /a n is tight (Athreya and Lahiri, 2006).…”

“…P ([−M, M ] p ) = 1, ∀P ∈ M, we first make standard assumptions that the data are i.i.d. with bounded components (Ben-David, 2007;Paul et al, 2021a).…”

“…Although often successful in practice, few of these approaches are grounded in rigorous theory or provide finite-sample statistical guarantees. The available consistency results are mostly asymptotic in nature Terada, 2014Terada, , 2015, lacking convergence rates or operating under restrictive assumptions on the relation between p and n (Paul et al, 2021a). For example, the recent large sample analysis (Paul et al, 2021a) of the hard Bregman k-means algorithm (Banerjee et al, 2005) obtains an asymptotic error rate of O( log n/n) under restrictive assumptions on the relation between n and p. The approach by Teboulle (2007) focuses on a unified framework from an optimization perspective, while Balcan et al (2008); Telgarsky and Dasgupta (2012) focus on different divergence-based methods to better understand the underlying feature-space.…”

Robust Linear Predictions: Analyses of Uniform Concentration, Fast Rates and Model Misspecification

“…This assumption is quite standard in literature (Klochkov et al, 2021;Chakraborty and Das, 2019). We note that we do not impose any boundedness assumption (Paul et al, 2021a;Chakraborty and Das, 2021) on the support of the underlying distribution.…”

Upper bound estimations of misclassification rate in the heteroscedastic clustering model with sub‐Gaussian noises

“…Its algorithms have gained enormous interests in biology, insurance industry, economic, statistics, geography and so on, so people put forward new ideas continually to improve or innovate the existing algorithms. For instance, recent advances in centre‐based clustering (Klochkov et al, 2021; Telgarsky & Dasgupta, 2013) try to conquer the drawbacks of the ‐means algorithm; clustering joint with Bregman divergence (Paul et al, 2021; Vellal et al, 2022) has been used to unify centroid‐based parametric clustering approaches. In addition, many researchers are also committed to studying related theory of clustering; see Bousquet and Zhivotovskiy (2021), Paul et al (2021), Klochkov et al (2021) and Vellal et al (2022).…”