DANCo: An intrinsic dimensionality estimator exploiting angle and norm concentration

Ceruti, Claudio; Bassis, Simone; Rozza, Alessandro; Lombardi, Gabriele; Casiraghi, Elena; Campadelli, Paola

doi:10.1016/j.patcog.2014.02.013

Cited by 70 publications

(96 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Hence, this process is implemented empirically by investigating the range of 12 ≤ k ≤ 52 with an interval 1. The parameter α controls the amount to which the class information should be incorporated [32]. Here, 0 ≤ α ≤ 1 with an interval 0.1 is performed.…”

Section: Experiments and Resultsmentioning

confidence: 99%

Correlation coefficient based supervised locally linear embedding for pulmonary nodule recognition

Panpan

Xia

2016

Computer Methods and Programs in Biomedicine

View full text Add to dashboard Cite

Background and Objective Dimensionality reduction techniques are developed to suppress the negative effects of high dimensional feature space of lung CT images on classification performance in computer aided detection (CAD) systems for pulmonary nodule detection. Methods An improved supervised locally linear embedding (SLLE) algorithm is proposed based on the concept of correlation coefficient. The Spearman’s rank correlation coefficient is introduced to adjust the distance metric in the SLLE algorithm to ensure that more suitable neighborhood points could be identified, and thus to enhance the discriminating power of embedded data. The proposed Spearman’s rank correlation coefficient based SLLE (SC2SLLE) is implemented and validated in our pilot CAD system using a clinical dataset collected from the publicly available lung image database consortium and image database resource initiative (LICD-IDRI). Particularly, a representative CAD system for solitary pulmonary nodule detection is designed and implemented. After a sequential medical image processing steps, 64 nodules and 140 non-nodules are extracted, and 34 representative features are calculated. The SC2SLLE, as well as SLLE and LLE algorithm are applied to reduce the dimensionality. Several quantitative measurements are also used to evaluate and compare the performance. Results Using a 5-fold cross-validation methodology, the proposed algorithm achieves 87.65% accuracy, 79.23% sensitivity, 91.43% specificity, and 8.57% false positive rate, on average. Experimental results indicate that the proposed algorithm outperforms the original locally linear embedding and SLLE coupled with the support vector machine (SVM) classifier. Conclusions Based on the preliminary results from a limited number of nodules in our dataset, this study demonstrates the great potential to improve the performance of a CAD system for nodule detection using the proposed SC2SLLE.

show abstract

Section: Experiments and Resultsmentioning

confidence: 99%

Correlation coefficient based supervised locally linear embedding for pulmonary nodule recognition

Panpan

Xia

2016

Computer Methods and Programs in Biomedicine

View full text Add to dashboard Cite

show abstract

“…3, we display the same analysis for an instance of the dataset C 6,12 (N = 2500), first introduced in [13] and considered a challenging dataset for ID estimation for its high intrinsic curvature. Although a thorough comparison of the performance of our algorithm with state-of-art ID estimators is out of the scope of our investigation (see [18] or [26] for nice recent meta-analysis), we observe that our prediction d 5.9 is pretty accurate (state-of-the-art estimators such as DANCo find d 6.9 on the C 6,12 dataset sampled in the same conditions).…”

Section: Multiscale Fci Estimatormentioning

confidence: 90%

“…CorrDim is very effective for the estimation of low IDs (d 10), whereas it systematically underestimates in the case of manifolds with larger IDs. This drawback is well known in the literature [18] and is only partially mitigated by more recent and advanced generalizations of CorrDim based on k-nearest-neighbors distances [12]. The reason why all these algorithms systematically fail for d 10 is due to a fundamental limitation of most geometric methods: indeed it is possible to prove [24] that the accurate estimation of the ID requires a number of samples N which grows exponentially in the intrinsic dimension d. As a consequence, one observes a systematic undersampling of the small radius region of the density of neighbors ρ X (r), as shown for the CorrDim estimator in the bottom left panel of Fig.…”

Section: Standard Algorithms For Ide and Extreme Locally Undersampledmentioning

confidence: 96%

“…Both mPCA and CorrDim, as well as their more recent generalizations such as DANCo [18], are based on the fundamental fact that, locally, samples in a dataset are effectively drawn uniformly from a d-dimensional disk {x x x ∈ R d : ||x|| ≤ 1} linearly embedded in R D . This is the informal way to state rigorous results based on the tangent space approximation to smooth manifolds and embeddings [18,25]. On one side, local neighbourhoods of large ID datasets (d 10) need a number of points N exponential in the ID to be sufficiently sampled; on the other hand, the tangent space approximation needs dense sampling of small patches of the manifold to be used in practice.…”

Section: Standard Algorithms For Ide and Extreme Locally Undersampledmentioning

confidence: 99%

See 1 more Smart Citation

Intrinsic dimension estimation for locally undersampled data

Erba

Rotondo

2019

Sci Rep

View full text Add to dashboard Cite

Identifying the minimal number of parameters needed to describe a dataset is a challenging problem known in the literature as intrinsic dimension estimation. All the existing intrinsic dimension estimators are not reliable whenever the dataset is locally undersampled, and this is at the core of the so called curse of dimensionality. Here we introduce a new intrinsic dimension estimator that leverages on simple properties of the tangent space of a manifold and extends the usual correlation integral estimator to alleviate the extreme undersampling problem. Based on this insight, we explore a multiscale generalization of the algorithm that is capable of (i) identifying multiple dimensionalities in a dataset, and (ii) providing accurate estimates of the intrinsic dimension of extremely curved manifolds. We test the method on manifolds generated from global transformations of high-contrast images, relevant for invariant object recognition and considered a challenge for state-of-the-art intrinsic dimension estimators.

show abstract

“…The idea behind local ID estimation is to operate at a scale where the manifold can be approximated by its tangent space [2]. The data contained in each neighbourhood is thus usually assumed to be uniformly distributed over an n-dimensional ball [13], [20]- [22]. In practice, ID proves sensitive to scale and finding an adequate neighbourhood size can be difficult, as it requires finding a trade-off between opposite requirements [1], [24].…”

Section: Dimensionalitymentioning

confidence: 99%

Estimating the effective dimension of large biological datasets using Fisher separability analysis

Albergante

Bac

Zinovyev

2019

2019 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

Modern large-scale datasets are frequently said to be high-dimensional. However, their data point clouds frequently possess structures, significantly decreasing their intrinsic dimensionality (ID) due to the presence of clusters, points being located close to low-dimensional varieties or fine-grained lumping. We test a recently introduced dimensionality estimator, based on analysing the separability properties of data points, on several benchmarks and real biological datasets. We show that the introduced measure of ID has performance competitive with state-of-the-art measures, being efficient across a wide range of dimensions and performing better in the case of noisy samples. Moreover, it allows estimating the intrinsic dimension in situations where the intrinsic manifold assumption is not valid.

show abstract

DANCo: An intrinsic dimensionality estimator exploiting angle and norm concentration

Cited by 70 publications

References 39 publications

Correlation coefficient based supervised locally linear embedding for pulmonary nodule recognition

Correlation coefficient based supervised locally linear embedding for pulmonary nodule recognition

Intrinsic dimension estimation for locally undersampled data

Estimating the effective dimension of large biological datasets using Fisher separability analysis

Contact Info

Product

Resources

About