Efficient Fine-Tuning of Deep Neural Networks with Effective Parameter Allocation

Abstract

It's commonplace in modern deep learning to achieve SOTA performance by fine-tuning a large, pretrained base model. Recent successes in natural language processing, attributed in part to knowledge transfer from large, pretrained, transformer-based language models, have sparked a similar revolution in computer vision via the introduction of Vision Transformers. As modern deep neural networks increase in performance, they also tend to increase in size. Key issues associated with fine-tuning such enormous models include storage overhead, as well as memory and / or latency requirements. Parameter efficient fine-tuning is a fairly recent paradigm which has been evolving alongside massive neural networks in part to address these issues. We showcase the effectiveness of parameter efficient fine-tuning of vision transformers, and introduce a simple yet effective method for learning a non-uniform parameter allocation given a fixed budget. We demonstrate our approach across a range of benchmark tasks in image classification and semantic segmentation.