General stochastic separation theorems with optimal bounds

Grechuk, Bogdan; Gorban, Alexander N.; Tyukin, Ivan

doi:10.1016/j.neunet.2021.01.034

Cited by 18 publications

(42 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It means that the simple classical techniques like linear Fisher's discriminant become unexpectedly powerful in high dimensions under some assumptions about regularity of probability distributions [12][13][14]. These assumptions can be rather mild and typically include absence of extremely dense lumps that are areas with relatively low volume but unexpectedly high probability (for more detail we refer to [15]). These lumps correspond to the narrow but high peaks of probability density.…”

Section: One-and Few-short Learningmentioning

confidence: 99%

“…2. In the situation of 'blessing of dimensionality', with sufficiently regular probability distribution in high dimensions the simple linear (or kernel [18]) one-and few-short methods become effective [7,14,15]. 3.…”

Section: One-and Few-short Learningmentioning

confidence: 99%

“…In a series of previous works, we focused on the second item [6,[11][12][13][14][15]25]. The blessing of dimensionality effects that make the one-and few-shot learning possible for regular distributions of data are based on the stochastic separation theorems.…”

Section: One-and Few-short Learningmentioning

confidence: 99%

“…Of course, the general theory and algorithms are focused on the universal part of the correctors. For many classical families of data distributions, it is proved that the well-known Fisher discriminant is surprisingly a powerful tool for constructing correctors if the dimension of the data space is sufficiently high (most results of this type are collected in [15]). This is proven for a wide class of distributions, including log-concave distributions, their convex combinations, and product distributions with bounded support.…”

Section: Internal Signals Inputs Outputsmentioning

confidence: 99%

See 3 more Smart Citations

High-Dimensional Separability for One- And Few-Shot Learning

Gorban

Grechuk

Mirkes

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

This work is driven by a practical question, corrections of Artificial Intelligence (AI) errors. Systematic re-training of a large AI system is hardly possible. To solve this problem, special external devices, correctors, are developed. They should provide quick and non-iterative system fix without modification of a legacy AI system. A common universal part of the AI corrector is a classifier that should separate undesired and erroneous behavior from normal operation. Training of such classifiers is a grand challenge at the heart of the one- and few-shot learning methods. Effectiveness of one- and few-short methods is based on either significant dimensionality reductions or the blessing of dimensionality effects. Stochastic separability is a blessing of dimensionality phenomenon that allows one-and few-shot error correction: in high-dimensional datasets under broad assumptions each point can be separated from the rest of the set by simple and robust linear discriminant. The hierarchical structure of data universe is introduced where each data cluster has a granular internal structure, etc. New stochastic separation theorems for the data distributions with fine-grained structure are formulated and proved. Separation theorems in infinite-dimensional limits are proven under assumptions of compact embedding of patterns into data space. New multi-correctors of AI systems are presented and illustrated with examples of predicting errors and learning new classes of objects by a deep convolutional neural network.

show abstract

Section: One-and Few-short Learningmentioning

confidence: 99%

Section: One-and Few-short Learningmentioning

confidence: 99%

Section: One-and Few-short Learningmentioning

confidence: 99%

Section: Internal Signals Inputs Outputsmentioning

confidence: 99%

See 2 more Smart Citations

High-Dimensional Separability for One- And Few-Shot Learning

Gorban

Grechuk

Mirkes

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…In the last decade, many other examples of blessing-of-dimensionality appeared, both in the general analysis of complex systems (see, e.g., [ 1 , 3 , 4 , 5 , 6 , 7 ]) and, specifically, in the analysis of neural networks; see, e.g., [ 8 , 9 , 10 , 11 ].…”

Section: Introduction: From Curse Of Dimensionality To Blessing Of Dimensionalitymentioning

confidence: 99%

Limit Theorems as Blessing of Dimensionality: Neural-Oriented Overview

Kreinovich

Kosheleva

2021

Entropy

View full text Add to dashboard Cite

As a system becomes more complex, at first, its description and analysis becomes more complicated. However, a further increase in the system’s complexity often makes this analysis simpler. A classical example is Central Limit Theorem: when we have a few independent sources of uncertainty, the resulting uncertainty is very difficult to describe, but as the number of such sources increases, the resulting distribution gets close to an easy-to-analyze normal one—and indeed, normal distributions are ubiquitous. We show that such limit theorems often make analysis of complex systems easier—i.e., lead to blessing of dimensionality phenomenon—for all the aspects of these systems: the corresponding transformation, the system’s uncertainty, and the desired result of the system’s analysis.

show abstract

Probability and Statistics

Grechuk

2021

Landscape of 21st Century Mathematics

View full text Add to dashboard Cite

General stochastic separation theorems with optimal bounds

Cited by 18 publications

References 27 publications

High-Dimensional Separability for One- And Few-Shot Learning

High-Dimensional Separability for One- And Few-Shot Learning

Limit Theorems as Blessing of Dimensionality: Neural-Oriented Overview

Probability and Statistics

Contact Info

Product

Resources

About