Local Intrinsic Dimensionality Based Features for Clustering

Abstract

One of the fundamental tasks of unsupervised learning is dataset clustering, to partition the input dataset into clusters composed by somehow "similar" objects that "differ" from the objects belonging to other classes. To this end, in this paper we assume that the different clusters are drawn from different, possibly intersecting, geometrical structures represented by manifolds embedded into a possibly higher dimensional space. Under these assumptions, and considering that each manifold is typified by a geometrical structure characterized by its intrinsic dimensionality, which (possibly) differs from the intrinsic dimensionalities of other manifolds, we code the input data by means of local intrinsic dimensionality estimates and features related to them, and we subsequently apply simple and basic clustering algorithms, since our interest is specifically aimed at assessing the discriminative power of the proposed features. Indeed, their encouraging discriminative quality is shown by a feature relevance test, by the clustering results achieved on both synthetic and real datasets, and by their comparison to those obtained by related and classical state-of-the-art clustering approaches.

Data segmentation based on the local intrinsic dimension

“…The idea that ID may vary in the same data is not new. In fact, many works have discussed the possibility of a variable ID and developed methods to estimate multiple IDs [7][8][9][10][11][12][13][14][15][16][17][18][20][21][22] . Our method builds on these previous contributions but is designed with the specific goal of overcoming technical limitations of other available approaches and make ID-based segmentation a general-purpose tool.…”

“…Let x = {x i } be i.i.d samples from a density ρ(x) with support on the union of K manifolds with varying dimensions. This multi-manifold framework is common with many previous works investigating heterogeneous dimension in a dataset [8][9][10][11][14][15][16][18][19][20]24,25 . Formally, let ρ(x) = K k=1 p k ρ k (x) where each ρ k (x) has support on a manifold of dimension d k and p .…”

“…Assumption i) is shared all methods that infer the ID from the distribution of distances (e.g., Refs. 5,6,[9][10][11][12]14 . Assumption ii) is the key assumption of TWO-NN: while of course it may not be perfectly satisfied, other methods used to infer the ID from distances require uniformity of the density on a larger scale including the first k neighbors, usually with k ≫ 2 .…”

Intrinsic Dimension Estimation: Relevant Techniques and a Benchmark Framework

“…Moreover, in [17] the authors mark that, in order to balance a classifier generalization ability and its empirical error, the complexity of the classification model should also be related to the id of the available dataset. Furthermore, since complex objects can be considered as structures composed by multiple manifolds that must be clustered to be processed separately, the knowledge of the local ids characterizing the considered object is fundamental to obtain a proper clustering [18].…”

Local Intrinsic Dimensionality, Entropy and Statistical Divergences

“…Local intrinsic dimensionality focuses on the intrinsic dimension of a particular query point and has been used in a range of applications. These include modeling deformation in granular materials [ 20 , 21 ], climate science [ 22 , 23 ], dimension reduction via local PCA [ 24 ], similarity search [ 25 ], clustering [ 26 ], outlier detection [ 27 ], statistical manifold learning [ 28 ], adversarial example detection [ 29 ], adversarial nearest neighbor characterization [ 30 , 31 ] and deep learning understanding [ 32 , 33 ]. In deep learning, it has been shown that adversarial examples are associated with high LID estimates, a characteristic that can be leveraged to build accurate adversarial example detectors [ 29 ].…”