Scaling Laws for the Few-Shot Adaptation of Pre-trained Image Classifiers

Abstract

Empirical science of neural scaling laws is a rapidly growing area of significant importance to the future of machine learning, particularly in the light of recent breakthroughs achieved by large-scale pre-trained models such as GPT-3, CLIP and DALL-e. Accurately predicting the neural network performance with increasing resources such as data, compute and model size provides a more comprehensive evaluation of different approaches across multiple scales, as opposed to traditional point-wise comparisons of fixed-size models on fixed-size benchmarks, and, most importantly, allows for focus on the best-scaling, and thus most promising in the future, approaches. In this work, we consider a challenging problem of few-shot learning in image classification, especially when the target data distribution in the few-shot phase is different from the source, training, data distribution, in a sense that it includes new image classes not encountered during training. Our current main goal is to investigate how the amount of pre-training data affects the few-shot generalization performance of standard image classifiers. Our key observations are that (1) such performance improvements are well-approximated by power laws (linear log-log plots) as the training set size increases, (2) this applies to both cases of target data coming from either the same or from a different domain (i.e., new classes) as the training data, and (3) few-shot performance on new classes converges at a faster rate than the standard classification performance on previously seen classes. Our findings shed new light on the relationship between scale and generalization.

Predictability and Surprise in Large Generative Models

“…Subsequent work has shown that similar scaling laws exist in generative models for other modalities (e.g., images, video, math, etc.) [35], audition [21], transfer from text to programming [36], few-shot adaptation of vision models [54], and more.…”

Neurodynamical Computing at the Information Boundaries of Intelligent Systems

“…This view might be summarized as the hope that defining features of biological intelligence will be inevitably unlocked by learning network architectures with the right connectional parameters at a sufficient scale of compute (i.e., proportional to input, model, and training sizes) to be attained in the future. Despite increasingly impressive capabilities, the latest and largest models, like GPT-3, OPT-175, DALL-E 2, and a recent "generalist" model called Gato [1][2][3][4][5][6], continue a long history of moving the goalposts for these biological features [7][8][9][10][11], which include abstraction and generalization; systematic compositionality of semantic relations; continual learning and causal inference; as well as the persistent ordersof-magnitude performance gaps reflected by low samplecomplexity and high energy-efficiency. Instead of presuming that scaling beyond trillion parameter models will yield intelligent machines from data soup, we infer that understanding biological intelligence sufficiently to formalize its defining features likely depends on characterizing fundamental mechanisms of brain computation and, importantly, solving how animals use their brains to construct subjective meaning from information [12][13][14][15][16][17][18][19].…”