Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark

Dumoulin, Vincent; Houlsby, Neil; Evci, Utku; Zhai, Xiaohua; Goroshin, Ross; Gelly, Sylvain; Larochelle, Hugo

doi:10.48550/arxiv.2104.02638

Cited by 8 publications

(26 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One prominent example is where they show that GPT-3, which is a large transformer model [Vaswani et al, 2017] trained on a large corpus of data, achieves substantial performance on many natural language processing (NLP) tasks and benchmarks in few-shot settings. On image recognition tasks, training on Instagram images [Mahajan et al, 2018] and JFT-300 [Sun et al, 2017] has been proven to be very effective in transfer and few-shot settings [Goyal et al, 2021, Pham et al, 2020, Dosovitskiy et al, 2020, Dumoulin et al, 2021]. Even when no example is provided (zero-shot), CLIP [Radford et al, 2021], which consists of a pair of image encoder and text encoder models trained with a contrastive loss on 400 million image-text pairs from the internet, can achieve remarkable performance.…”

Section: Introductionmentioning

confidence: 99%

Exploring the Limits of Large Scale Pre-training

Abnar,

Dehghani,

Neyshabur

et al. 2021

Preprint

View full text Add to dashboard Cite

Recent developments in large-scale machine learning suggest that by scaling up data, model size and training time properly, one might observe that improvements in pre-training would transfer favorably to most downstream tasks. In this work, we systematically study this phenomena and establish that, as we increase the upstream accuracy, the performance of downstream tasks saturates. In particular, we investigate more than 4800 experiments on Vision Transformers, MLP-Mixers and ResNets with number of parameters ranging from ten million to ten billion, trained on the largest scale of available image data (JFT, ImageNet21K) and evaluated on more than 20 downstream image recognition tasks. We propose a model for downstream performance that reflects the saturation phenomena and captures the nonlinear relationship in performance of upstream and downstream tasks. Delving deeper to understand the reasons that give rise to these phenomena, we show that the saturation behavior we observe is closely related to the way that representations evolve through the layers of the models. We showcase an even more extreme scenario where performance on upstream and downstream are at odds with each other. That is, to have a better downstream performance, we need to hurt upstream accuracy.

show abstract

Section: Introductionmentioning

confidence: 99%

Exploring the Limits of Large Scale Pre-training

Abnar,

Dehghani,

Neyshabur

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…If N τ is large (e.g. the recent VTAB+MD benchmark [11] requires a task's support set to be as large as 1000 images), memory on a single GPU is thus quickly exceeded for large images.…”

Section: Our Contributionsmentioning

confidence: 99%

“…query example. In some cases, this can be as many as 1000 images [11]. As a result, the amount of memory required for the computational graph grows linearly with the number of support images, and quadratically with their dimension.…”

Section: Introductionmentioning

confidence: 99%

“…As a result, the amount of memory required for the computational graph grows linearly with the number of support images, and quadratically with their dimension. In contrast, transfer learning approaches can employ standard batch processing techniques to scale to larger images when under memory constraints -a feature which has contributed significantly to their recent success on few-shot benchmarks [11].…”

Section: Introductionmentioning

confidence: 99%

“…We use LITE to train key meta-learning methods and show that our best performing instantiation -Simple CNAPs [5] with LITE -achieves state-of-the-art results relative to all meta-learners on two challenging few-shot benchmarks: VTAB+MD [11], an extensive suite of both meta-learning and transfer learning tasks, and ORBIT [14], an object recognition benchmark of high-variation real-world videos. Our results showcase the unique advantage of meta-learning methods -that when properly trained they can be competitive with transfer learning approaches in terms of accuracy for a fraction of the computational cost at test time -and they serve as a counterpoint to the recent narrative that transfer learning is all you need for few-shot classification.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Memory Efficient Meta-Learning with Large Images

Bronskill¹,

Massiceti²,

Patacchiola³

et al. 2021

Preprint

View full text Add to dashboard Cite

Meta learning approaches to few-shot classification are computationally efficient at test time requiring just a few optimization steps or single forward pass to learn a new task, but they remain highly memory-intensive to train. This limitation arises because a task's entire support set, which can contain up to 1000 images, must be processed before an optimization step can be taken. Harnessing the performance gains offered by large images thus requires either parallelizing the meta-learner across multiple GPUs, which may not be available, or trade-offs between task and image size when memory constraints apply. We improve on both options by proposing LITE, a general and memory efficient episodic training scheme that enables meta-training on large tasks composed of large images on a single GPU. We achieve this by observing that the gradients for a task can be decomposed into a sum of gradients over the task's training images. This enables us to perform a forward pass on a task's entire training set but realize significant memory savings by back-propagating only a random subset of these images which we show is an unbiased approximation of the full gradient. We use LITE to train meta-learners and demonstrate new state-of-the-art accuracy on the real-world ORBIT benchmark and 3 of the 4 parts of the challenging VTAB+MD benchmark relative to leading meta-learners. LITE also enables meta-learners to be competitive with transfer learning approaches but at a fraction of the test-time computational cost, thus serving as a counterpoint to the recent narrative that transfer learning is all you need for few-shot classification.

show abstract

A Comparison of Machine Learning Methods for Cross-Domain Few-Shot Learning

Wang

Gouk

Frank

et al. 2020

AI 2020: Advances in Artificial Intelligence

View full text Add to dashboard Cite

We present an empirical evaluation of machine learning algorithms in cross-domain few-shot learning based on a fixed pre-trained feature extractor. Experiments were performed in five target domains (CropDisease, EuroSAT, Food101, ISIC and ChestX) and using two feature extractors: a ResNet10 model trained on a subset of ImageNet known as miniImageNet and a ResNet152 model trained on the ILSVRC 2012 subset of ImageNet. Commonly used machine learning algorithms including logistic regression, support vector machines, random forests, nearest neighbour classification, naïve Bayes, and linear and quadratic discriminant analysis were evaluated on the extracted feature vectors. We also evaluated classification accuracy when subjecting the feature vectors to normalisation using p-norms. Algorithms originally developed for the classification of gene expression data-the nearest shrunken centroid algorithm and LDA ensembles obtained with random projections-were also included in the experiments, in addition to a cosine similarity classifier that has recently proved popular in few-shot learning. The results enable us to identify algorithms, normalisation methods and pre-trained feature extractors that perform well in cross-domain few-shot learning. We show that the cosine similarity classifier and 2-regularised 1-vs-rest logistic regression are generally the best-performing algorithms. We also show that algorithms such as LDA yield consistently higher accuracy when applied to 2-normalised feature vectors. In addition, all classifiers generally perform better when extracting feature vectors using the ResNet152 model instead of the ResNet10 model.

show abstract

Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark

Cited by 8 publications

References 28 publications

Exploring the Limits of Large Scale Pre-training

Exploring the Limits of Large Scale Pre-training

Memory Efficient Meta-Learning with Large Images

A Comparison of Machine Learning Methods for Cross-Domain Few-Shot Learning

Contact Info

Product

Resources

About