BinPlay: A Binary Latent Autoencoder for Generative Replay Continual Learning

Deja, Kamil Rafał; Wawrzyński, Paweł; Marczak, Daniel; Masarczyk, Wojciech; Trzciński, Tomasz Piotr

doi:10.48550/arxiv.2011.14960

Cited by 2 publications

(4 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this section, the proposed HyBNN and FedHyBNN are tested on the MNIST dataset. For the experiment, the VAE of HyBNN follows the same architecture as BinPlay Deja et al [2020], the encoder consists of three convolutional layers with batch normalization followed by an output layer consisting of 200 neurons. The binary latent space consists of merely 200 bits or 25 bytes.…”

Section: Resultsmentioning

confidence: 99%

“…It has been successfully applied on images Carreira-Perpinán and Raziperchikolaei [2015] and on videos Song et al [2018] to produce state-of-the-art results. Recently, binary autoencoders have been used for continual learning, specifically Deja et al Deja et al [2020] used them to create unique sequential binary codes for each input sample, similar to hashing and then using older hashes for generative replay to solve the problem of catastrophic forgetting.…”

Section: Related Workmentioning

confidence: 99%

“…The extracted features are then used as the input for the Binary Neural Network. To solve the problem of input binarization loss we use the Variational Autoencoder with a binary latent space as described in BinPlay Deja et al [2020] to extract the binary representation of the input. This input can then be directly fed to the binary neural network, bypassing the initial binarization step entirely.…”

Section: Input Binarization Is a Major Source Of Information Loss And...mentioning

confidence: 99%

See 2 more Smart Citations

HyBNN and FedHyBNN: (Federated) Hybrid Binary Neural Networks

Dua¹

2022

Preprint

View full text Add to dashboard Cite

Binary Neural Networks (BNNs), neural networks with weights and activations constrained to -1(0) and +1, are an alternative to deep neural networks which offer faster training, lower memory consumption and lightweight models, ideal for use in resource constrained devices while being able to utilize the architecture of their deep neural network counterpart. However, the input binarization step used in BNNs causes a severe accuracy loss. In this paper, we introduce a novel hybrid neural network architecture, Hybrid Binary Neural Network (HyBNN), consisting of a task-independent, general, full-precision variational autoencoder with a binary latent space and a task specific binary neural network that is able to greatly limit the accuracy loss due to input binarization by using the full precision variational autoencoder as a feature extractor. We use it to combine the state-of-the-art accuracy of deep neural networks with the much faster training time, quicker test-time inference and power efficiency of binary neural networks. We show that our proposed system is able to very significantly outperform a vanilla binary neural network with input binarization. We also introduce FedHyBNN, a highly communication efficient federated counterpart to HyBNN and demonstrate that it is able to reach the same accuracy as its non-federated equivalent. We make our source code, experimental parameters and models available at: https://anonymous.4open.science/r/HyBNN.Preprint. Under review.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Input Binarization Is a Major Source Of Information Loss And...mentioning

confidence: 99%

See 1 more Smart Citation

HyBNN and FedHyBNN: (Federated) Hybrid Binary Neural Networks

Dua¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…However, we condition our generative model on task number. For that purpose, we use binary encoding with co-prime numbers as proposed in [6].…”

Section: Evaluation Setupmentioning

confidence: 99%

Multiband VAE: Latent Space Partitioning for Knowledge Consolidation in Continual Learning

Deja¹,

Wawrzyński²,

Masarczyk³

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

We propose a new method for unsupervised continual knowledge consolidation in generative models that relies on the partitioning of Variational Autoencoder's latent space. Acquiring knowledge about new data samples without forgetting previous ones is a critical problem of continual learning. Currently proposed methods achieve this goal by extending the existing model while constraining its behavior not to degrade on the past data, which does not exploit the full potential of relations within the entire training dataset. In this work, we identify this limitation and posit the goal of continual learning as a knowledge accumulation task. We solve it by continuously re-aligning latent space partitions that we call bands which are representations of samples seen in different tasks, driven by the similarity of the information they contain. In addition, we introduce a simple yet effective method for controlled forgetting of past data that improves the quality of reconstructions encoded in latent bands and a latent space disentanglement technique that improves knowledge consolidation. On top of the standard continual learning evaluation benchmarks, we evaluate our method on a new knowledge consolidation scenario and show that the proposed approach outperforms state-ofthe-art by up to twofold across all testing scenarios. 1

show abstract

BinPlay: A Binary Latent Autoencoder for Generative Replay Continual Learning

Cited by 2 publications

References 21 publications

HyBNN and FedHyBNN: (Federated) Hybrid Binary Neural Networks

HyBNN and FedHyBNN: (Federated) Hybrid Binary Neural Networks

Multiband VAE: Latent Space Partitioning for Knowledge Consolidation in Continual Learning

Contact Info

Product

Resources

About