Unravelling Small Sample Size Problems in the Deep Learning World

Abstract

The growth and success of deep learning approaches can be attributed to two major factors: availability of hardware resources and availability of large number of training samples. For problems with large training databases, deep learning models have achieved superlative performances. However, there are a lot of small sample size or S 3 problems for which it is not feasible to collect large training databases. It has been observed that deep learning models do not generalize well on S 3 problems and specialized solutions are required. In this paper, we first present a review of deep learning algorithms for small sample size problems in which the algorithms are segregated according to the space in which they operate, i.e. input space, model space, and feature space. Secondly, we present Dynamic Attention Pooling approach which focuses on extracting global information from the most discriminative sub-part of the feature map. The performance of the proposed dynamic attention pooling is analyzed with state-of-the-art ResNet model on relatively small publicly available datasets such as SVHN, C10, C100, and TinyImageNet.

Ten deep learning techniques to address small data problems with remote sensing

“…Finally, we offered a set of practical recommendations about how RS scientists can better implement DL techniques to fully take advantage of a small dataset. As one previous paper noted (Keshari et al, 2020) a variety of approaches can be used to solve the small data problem, such as data augmentation, data fine-tuning, the adaptation of pre-trained models, and reducing the dependence on large-sample learning. However, in our review, we also presented even more techniques that are worth considering when working with a small dataset.…”