Mixture-of-Variational-Experts for Continual Learning

Abstract

One significant shortcoming of machine learning is the poor ability of models to solve new problems quicker and without forgetting acquired knowledge. To better understand this issue, continual learning has emerged to systematically investigate learning protocols where the model sequentially observes samples generated by a series of tasks. First, we propose an optimality principle that facilitates a trade-off between learning and forgetting. We derive this principle from an information-theoretic formulation of bounded rationality and show its connections to other continual learning methods. Second, based on this principle, we propose a neural network layer for continual learning, called Mixture-of-Variational-Experts (MoVE), that alleviates forgetting while enabling the beneficial transfer of knowledge to new tasks. Our experiments on variants of the MNIST and CI-FAR10 datasets demonstrate the competitive performance of MoVE layers when compared to stateof-the-art approaches.

A Novel Transfer Learning Framework for Multimodal Skin Lesion Analysis

“…The ViT architecture, comprised of FF and self-attention layers, was developed to overcome the drawbacks of these approaches. The ViT architecture forms the basis of a visual transformer originally developed for NLP tasks [34]. The transformer's capability to interpret coincident data sequences simultaneously is one of its prime -advantages, coveted for CV applications [35].…”