Deep Mixture Generative Autoencoders

Lecture Notes in Computer Science

2022

Self Cite

Due to their inference, data representation and reconstruction properties, Variational Autoencoders (VAE) have been successfully used in continual learning classification tasks. However, their ability to generate images with specifications corresponding to the classes and databases learned during Continual Learning (CL) is not well understood and catastrophic forgetting remains a significant challenge. In this paper, we firstly analyze the forgetting behaviour of VAEs by developing a new theoretical framework that formulates CL as a dynamic optimal transport problem. This framework proves approximate bounds to the data likelihood without requiring the task information and explains how the prior knowledge is lost during the training process. We then propose a novel memory buffering approach, namely the Online Cooperative Memorization (OCM) framework, which consists of a Short-Term Memory (STM) that continually stores recent samples to provide future information for the model, and a Long-Term Memory (LTM) aiming to preserve a wide diversity of samples. The proposed OCM transfers certain samples from STM to LTM according to the information diversity selection criterion without requiring any supervised signals. The OCM framework is then combined with a dynamic VAE expansion mixture network for further enhancing its performance.

Section: Evaluation Of the Reconstruction Qualitymentioning

confidence: 99%

“…We follow the settings from [61], as described in Appendix-H.3 from SM, after resizing all images to 64 × 64 pixels. The FID and IS results are provided in Table 5 and the results of all baselines (training on a single dataset) are cited from [61]. The visual results are shown in Fig.…”

Section: Evaluation Of the Reconstruction Qualitymentioning

confidence: 99%

Continual Variational Autoencoder Learning via Online Cooperative Memorization

Lecture Notes in Computer Science

2022

Self Cite

“…The primary drawback of these models is that most existing DAMs would only focus on classification tasks and can not learn meaningful representations across domains under the unsupervised learning setting. Furthermore, there were some attempts for using mixture models to learn complex datasets [38], [39], [40] or learn an infinite number of tasks [12], [18], [41], [42]. However, the expansion of these models relies on the estimation of the sample log-likelihood, which requires each expert to have an explicit probabilistic function form.…”

Section: Dynamic Architecturesmentioning

confidence: 99%

Dynamic Self-Supervised Teacher-Student Network Learning

IEEE Trans. Pattern Anal. Mach. Intell.

2022

Lifelong learning (LLL) represents the ability of an artificial intelligence system to learn successively a sequence of different databases. In this paper we introduce the Dynamic Self-Supervised Teacher-Student Network (D-TS), representing a more general LLL framework, where the Teacher is implemented as a dynamically expanding mixture model which automatically increases its capacity to deal with a growing number of tasks. We propose the Knowledge Discrepancy Score (KDS) criterion for measuring the relevance of the incoming information characterizing a new task when compared to the existing knowledge accumulated by the Teacher module from its previous training. The KDS ensures a light Teacher architecture while also enabling to reuse the learned knowledge whenever appropriate, accelerating the learning of given tasks. The Student module is implemented as a lightweight probabilistic generative model. We introduce a novel self-supervised learning for the Student that allows to capture cross-domain latent representations from the entire knowledge accumulated by the Teacher as well as from novel data. We perform several experiments which show that D-TS can achieve the state of the art results in LLL while requiring fewer parameters than other methods.

“…In this paper we consider a model made up of a mixture of networks [45] which is able to deal with three different learning scenarios: supervised, semi-supervised and unsupervised, under the lifelong learning setting. Let us consider a sequence of tasks and denote…”

Section: The Lifelong Mixture Of Vaes a Problem Formulationmentioning

confidence: 99%

Lifelong Mixture of Variational Autoencoders

2021

Preprint

Self Cite

In this paper, we propose an end-to-end lifelong learning mixture of experts. Each expert is implemented by a Variational Autoencoder (VAE). The experts in the mixture system are jointly trained by maximizing a mixture of individual component evidence lower bounds (MELBO) on the loglikelihood of the given training samples. The mixing coefficients in the mixture model, control the contributions of each expert in the global representation. These are sampled from a Dirichlet distribution whose parameters are determined through nonparametric estimation during the lifelong learning. The model can learn new tasks fast when these are similar to those previously learnt. The proposed Lifelong mixture of VAE (L-MVAE) expands its architecture with new components when learning a completely new task. After the training, our model can automatically determine the relevant expert to be used when fed with new data samples. This mechanism benefits both the memory efficiency and the required computational cost as only one expert is used during the inference. The L-MVAE inference model is able to perform interpolations in the joint latent space across the data domains associated with different tasks and is shown to be efficient for disentangled learning representation. The code is available : https://github.com/dtuzi123/LifelongMixtureVAEs