The Remarkable Simplicity of Very High Dimensional Data: Application of Model-Based Clustering

Abstract

An ultrametric topology formalizes the notion of hierarchical structure. An ultrametric embedding, referred to here as ultrametricity, is implied by a hierarchical embedding. Such hierarchical structure can be global in the data set, or local. By quantifying extent or degree of ultrametricity in a data set, we show that ultrametricity becomes pervasive as dimensionality and/or spatial sparsity increases. This leads us to assert that very high dimensional data are of simple structure. We exemplify this finding through a range of simulated data cases. We discuss also application to very high frequency time series segmentation and modeling.

Model-based clustering of high-dimensional data: A review

“…Several authors, such as [44,75], have shown that, in the context of clustering, high-dimensional spaces do have useful characteristics which ease the classification of data in those spaces. In particular, Scott and Thompson [88] showed that high-dimensional spaces are mostly empty.…”

A Bayesian criterion for cluster stability

“…as an asymptotic lower bound on the first term of Eq. (24). Clearly, as ξ → 0, the first term in Eq.…”

High dimensional low sample size activity recognition using geometric classifiers

“…Affine Hulls, on the other hand, give a loose approximation of class regions as they model each class as an affine subspace. Therefore NAH classification may prove a promising instance based classifier in HDLSS settings where the notion of a neighborhood breaks down [9,11,17].…”

“…Murtagh [17] showed that ultrametricity becomes pervasive as dimensionality and spatial sparsity increases and used this property in model based clustering. Klement et al [15] proved that, for d → ∞, random and non-random scenarios are not distinguishable by any metric such that distances become approximately equal.…”