No abstract

No abstract

No abstract

Most deep neural networks use simple, fixed activation functions, such as sigmoids or rectified linear units, regardless of domain or network structure. We introduce differential equation units (DEUs), an improvement to modern neural networks, which enables each neuron to learn a particular nonlinear activation function from a family of solutions to an ordinary differential equation. Specifically, each neuron may change its functional form during training based on the behavior of the other parts of the network. We show that using neurons with DEU activation functions results in a more compact network capable of achieving comparable, if not superior, performance when compared to much larger networks.

It's commonplace in modern deep learning to achieve SOTA performance by fine-tuning a large, pretrained base model. Recent successes in natural language processing, attributed in part to knowledge transfer from large, pretrained, transformer-based language models, have sparked a similar revolution in computer vision via the introduction of Vision Transformers. As modern deep neural networks increase in performance, they also tend to increase in size. Key issues associated with fine-tuning such enormous models include storage overhead, as well as memory and / or latency requirements. Parameter efficient fine-tuning is a fairly recent paradigm which has been evolving alongside massive neural networks in part to address these issues. We showcase the effectiveness of parameter efficient fine-tuning of vision transformers, and introduce a simple yet effective method for learning a non-uniform parameter allocation given a fixed budget. We demonstrate our approach across a range of benchmark tasks in image classification and semantic segmentation.