Approximating Spectral Clustering via Sampling: A Review

Abstract

Spectral clustering refers to a family of unsupervised learning algorithms that compute a spectral embedding of the original data based on the eigenvectors of a similarity graph. This non-linear transformation of the data is both the key of these algorithms' success and their Achilles heel: forming a graph and computing its dominant eigenvectors can indeed be computationally prohibitive when dealing with more that a few tens of thousands of points. In this paper, we review the principal research efforts aiming to reduce this computational cost. We focus on methods that come with a theoretical control on the clustering performance and incorporate some form of sampling in their operation. Such methods abound in the machine learning, numerical linear algebra, and graph signal processing literature and, amongst others, include Nyström-approximation, landmarks, coarsening, coresets, and compressive spectral clustering. We present the approximation guarantees available for each and discuss practical merits and limitations. Surprisingly, despite the breadth of the literature explored, we conclude that there is still a gap between theory and practice: the most scalable methods are only intuitively motivated or loosely controlled, whereas those that come with end-to-end guarantees rely on strong assumptions or enable a limited gain of computation time.

Kernel Matrix Approximation on Class-Imbalanced Data With an Application to Scientific Simulation

“…The cluster assignments are disregarded in this case because k typically exceeds the actual number of classes or unique labels. Solving this optimization problem is NP-hard and the Lloyd-Max algorithm is a popular iterative technique for approximating the optimal solution [32]. However, this algorithm requires the entire data set X to be stored in the main memory and incurs a high computational cost, taking O(npm) time per iteration.…”

Scalable Spectral Clustering With Nyström Approximation: Practical and Theoretical Aspects

“…Various methods have been proposed to accelerate spectral clustering by computing an approximate spectral embedding of the original data. Recent work [23] presented an excellent review of the literature on this topic for interested readers. In this paper, we divide the related work into two main categories: (1) methods that circumvent the computation of the full kernel matrix, and (2) techniques that consider the similarity graph as one of the inputs to spectral clustering and, thus, ignore the cost associated with step 2 of Alg.…”

Sparse and smooth: Improved guarantees for spectral clustering in the dynamic stochastic block model

“…Beyond SC, many other methods have been proposed, such as Maximum Likelihood or variational approaches, which are consistent for the SBM and DSBM [6,30,29], Bayesian approaches [49], learning-based approaches [2], or neural networks [5]. Many variants of the SC itself exist, often to accelerate computation [41]. We focus here on the traditional SC.…”