High-Dimensional Separability for One- And Few-Shot Learning

Abstract

This work is driven by a practical question, corrections of Artificial Intelligence (AI) errors. Systematic re-training of a large AI system is hardly possible. To solve this problem, special external devices, correctors, are developed. They should provide quick and non-iterative system fix without modification of a legacy AI system. A common universal part of the AI corrector is a classifier that should separate undesired and erroneous behavior from normal operation. Training of such classifiers is a grand challenge at the heart of the one- and few-shot learning methods. Effectiveness of one- and few-short methods is based on either significant dimensionality reductions or the blessing of dimensionality effects. Stochastic separability is a blessing of dimensionality phenomenon that allows one-and few-shot error correction: in high-dimensional datasets under broad assumptions each point can be separated from the rest of the set by simple and robust linear discriminant. The hierarchical structure of data universe is introduced where each data cluster has a granular internal structure, etc. New stochastic separation theorems for the data distributions with fine-grained structure are formulated and proved. Separation theorems in infinite-dimensional limits are proven under assumptions of compact embedding of patterns into data space. New multi-correctors of AI systems are presented and illustrated with examples of predicting errors and learning new classes of objects by a deep convolutional neural network.

Learning from few examples with nonlinear feature maps

“…However, comprehensive theoretical justification of these schemes is yet to be seen. Recent work [7], [8] suggested a new framework offering a pathway for understanding of fewshot learning. Instead of focusing on classical ideas rooted in empirical risk minimisation coupled with distribution-agnostic bounds, it explores the interplay between the geometry of feature spaces and concentration of measure phenomena [9].…”

“…In this work we adopt the theoretical framework proposed in [7], [8] and generalise it beyond the original setting whereby the problem of few-shot learning is analysed in models' native feature spaces. Here we explore how the problem of few-shot learning changes if one allows a nonlinear transformation of these features.…”

Towards a mathematical understanding of learning from few examples with nonlinear feature maps

“…Although recent work [3] provided new relevant insights explaining the coexistence of both generalisation and overfitting, it does not address the challenge of learning from low volumes of data. Another relevant approach has been developed in [4], driven by a need to identify and correct errors made by modern high dimensional AI systems. Rather than retraining the underlying system, which may be prohibitively expensive and runs the risk of catastrophically forgetting previous training, the focus is on building simple auxiliary systems to correct [4] or add functionality to existing AI systems.…”

“…Another relevant approach has been developed in [4], driven by a need to identify and correct errors made by modern high dimensional AI systems. Rather than retraining the underlying system, which may be prohibitively expensive and runs the risk of catastrophically forgetting previous training, the focus is on building simple auxiliary systems to correct [4] or add functionality to existing AI systems. It has been proven under certain assumptions that this approach is effective: with high probability the preexisting knowledge of the underlying system is retained and utilised when appropriate, while the new functionality is effectively learned.…”