Limit Theorems as Blessing of Dimensionality: Neural-Oriented Overview

Kreinovich, Vladik; Kosheleva, Olga

doi:10.3390/e23050501

Cited by 7 publications

(6 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In particular, the blessing of dimensionality is closely connected to the various versions of the central Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 30 June 2021 doi:10.20944/preprints202106.0718.v1 limit theorem in probability theory [19]. Of course, there remain open questions about sharp estimates for some distribution classes, but the general picture seems to be clear now.…”

Section: One-and Few-short Learningmentioning

confidence: 99%

High-Dimensional Separability for One- And Few-Shot Learning

Gorban

Grechuk

Mirkes

et al. 2021

Preprint

View full text Add to dashboard Cite

This work is driven by a practical question, corrections of Artificial Intelligence (AI) errors. Systematic re-training of a large AI system is hardly possible. To solve this problem, special external devices, correctors, are developed. They should provide quick and non-iterative system fix without modification of a legacy AI system. A common universal part of the AI corrector is a classifier that should separate undesired and erroneous behavior from normal operation. Training of such classifiers is a grand challenge at the heart of the one- and few-shot learning methods. Effectiveness of one- and few-short methods is based on either significant dimensionality reductions or the blessing of dimensionality effects. Stochastic separability is a blessing of dimensionality phenomenon that allows one-and few-shot error correction: in high-dimensional datasets under broad assumptions each point can be separated from the rest of the set by simple and robust linear discriminant. The hierarchical structure of data universe is introduced where each data cluster has a granular internal structure, etc. New stochastic separation theorems for the data distributions with fine-grained structure are formulated and proved. Separation theorems in infinite-dimensional limits are proven under assumptions of compact embedding of patterns into data space. New multi-correctors of AI systems are presented and illustrated with examples of predicting errors and learning new classes of objects by a deep convolutional neural network.

show abstract

Section: One-and Few-short Learningmentioning

confidence: 99%

High-Dimensional Separability for One- And Few-Shot Learning

Gorban

Grechuk

Mirkes

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…These blessing of dimensionality phenomena are closely connected to the concentration of measure [26][27][28][29]. In particular, the blessing of dimensionality is closely connected to the various versions of the central limit theorem in probability theory [19]. Of course, there remain open questions about sharp estimates for some distribution classes, but the general picture seems to be clear now.…”

Section: One-and Few-short Learningmentioning

confidence: 99%

“…In proof of Theorem 3 we construct explicit estimates of probability in (19). This construction (eq.…”

Section: Granular Models Of Clustersmentioning

confidence: 99%

High-dimensional separability for one- and few-shot learning

Gorban,

Grechuk,

Mirkes

et al. 2021

Preprint

View full text Add to dashboard Cite

This work is driven by a practical question, corrections of Artificial Intelligence (AI) errors. Systematic re-training of a large AI system is hardly possible. To solve this problem, special external devices, correctors, are developed. They should provide quick and non-iterative system fix without modification of a legacy AI system. A common universal part of the AI corrector is a classifier that should separate undesired and erroneous behavior from normal operation. Training of such classifiers is a grand challenge at the heart of the one-and few-shot learning methods. Effectiveness of one-and few-short methods is based on either significant dimensionality reductions or the blessing of dimensionality effects. Stochastic separability is a blessing of dimensionality phenomenon that allows one-and few-shot error correction: in high-dimensional datasets under broad assumptions each point can be separated from the rest of the set by simple and robust linear discriminant. The hierarchical structure of data universe is introduced where each data cluster has a granular internal structure, etc. New stochastic separation theorems for the data distributions with fine-grained structure are formulated and proved. Separation theorems in infinite-dimensional limits are proven under assumptions of compact embedding of patterns into data space. New multi-correctors of AI systems are presented and illustrated with examples of predicting errors and learning new classes of objects by a deep convolutional neural network.

show abstract

“…All these theorems have a similar structure: for large dimensions, even in an exponentially large (relatively to the dimension) set of points, each point is separable from the rest by a linear functional, which is given by a simple explicit formula. These blessings of dimensionality phenomena are closely connected to the concentration of measure [44][45][46][47][48] and to the various versions of the central limit theorem in probability theory [49]. Of course, there remain open questions about sharp estimates for some distribution classes, but the general picture seems to be clear now.…”

mentioning

confidence: 99%

“…Two modern books include most of the classical results and many new achievements of concentration of measure phenomena needed in data science [44,45] (but they do not include new stochastic separation theorems). Links between the blessing of dimensionality and the classical central limit theorems are recently discussed in [49].…”

mentioning

confidence: 99%

High-Dimensional Separability for One- and Few-Shot Learning

Gorban

Grechuk

Mirkes

et al. 2021

Entropy

View full text Add to dashboard Cite

This work is driven by a practical question: corrections of Artificial Intelligence (AI) errors. These corrections should be quick and non-iterative. To solve this problem without modification of a legacy AI system, we propose special `external’ devices, correctors. Elementary correctors consist of two parts, a classifier that separates the situations with high risk of error from the situations in which the legacy AI system works well and a new decision that should be recommended for situations with potential errors. Input signals for the correctors can be the inputs of the legacy AI system, its internal signals, and outputs. If the intrinsic dimensionality of data is high enough then the classifiers for correction of small number of errors can be very simple. According to the blessing of dimensionality effects, even simple and robust Fisher’s discriminants can be used for one-shot learning of AI correctors. Stochastic separation theorems provide the mathematical basis for this one-short learning. However, as the number of correctors needed grows, the cluster structure of data becomes important and a new family of stochastic separation theorems is required. We refuse the classical hypothesis of the regularity of the data distribution and assume that the data can have a rich fine-grained structure with many clusters and corresponding peaks in the probability density. New stochastic separation theorems for data with fine-grained structure are formulated and proved. On the basis of these theorems, the multi-correctors for granular data are proposed. The advantages of the multi-corrector technology were demonstrated by examples of correcting errors and learning new classes of objects by a deep convolutional neural network on the CIFAR-10 dataset. The key problems of the non-classical high-dimensional data analysis are reviewed together with the basic preprocessing steps including the correlation transformation, supervised Principal Component Analysis (PCA), semi-supervised PCA, transfer component analysis, and new domain adaptation PCA.

show abstract

Limit Theorems as Blessing of Dimensionality: Neural-Oriented Overview

Cited by 7 publications

References 39 publications

High-Dimensional Separability for One- And Few-Shot Learning

High-Dimensional Separability for One- And Few-Shot Learning

High-dimensional separability for one- and few-shot learning

High-Dimensional Separability for One- and Few-Shot Learning

Contact Info

Product

Resources

About