Intrinsic dimension estimation based on local adjacency information

Qiu, Haiquan; Yang, Youlong; Li, Benchong

doi:10.1016/j.ins.2021.01.017

Cited by 5 publications

(2 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Following 20 , we consider the MNIST (focusing on the training points representing digit 1: ) and the Isolet datasets ( ). Moreover, we consider the Isomap faces dataset ( ) as in 33 , 43 , and the CIFAR-10 dataset as in 44 (training data, ).…”

Section: Resultsmentioning

confidence: 99%

“…The geometric properties of a dataset are also exploited by the ESS estimator 31 , which is based on the evaluation of simplex volumes spanned by data points. Finally, Serra and Mandjes 32 and Qiu et al 33 estimated the id via random graph models applied to the adjacency matrices among data points, recovered by connecting observations whose distances do not exceed a certain threshold.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The generalized ratios intrinsic dimension estimator

Denti

Doimo

Laio

et al. 2022

Sci Rep

View full text Add to dashboard Cite

Modern datasets are characterized by numerous features related by complex dependency structures. To deal with these data, dimensionality reduction techniques are essential. Many of these techniques rely on the concept of intrinsic dimension (), a measure of the complexity of the dataset. However, the estimation of this quantity is not trivial: often, the depends rather dramatically on the scale of the distances among data points. At short distances, the can be grossly overestimated due to the presence of noise, becoming smaller and approximately scale-independent only at large distances. An immediate approach to examining the scale dependence consists in decimating the dataset, which unavoidably induces non-negligible statistical errors at large scale. This article introduces a novel statistical method, , that allows estimating the as an explicit function of the scale without performing any decimation. Our approach is based on rigorous distributional results that enable the quantification of uncertainty of the estimates. Moreover, our method is simple and computationally efficient since it relies only on the distances among data points. Through simulation studies, we show that is asymptotically unbiased, provides comparable estimates to other state-of-the-art methods, and is more robust to short-scale noise than other likelihood-based approaches.

show abstract