A Binary Variational Autoencoder for Hashing

Mena, Francisco; Ñanculef, Ricardo

doi:10.1007/978-3-030-33904-3_12

Cited by 7 publications

(14 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As in related works [17], we pose hashing as an inference problem, where the objective is to learn a probability distribution q φ (b|x) of the code b ∈ {0, 1} B corresponding to an input pattern x. This framework is based on a generative process involving two steps: (i) choose an entry of the hash table according to some probability distribution p θ (b), and (ii) sample an observation x indexed by that address according to a conditional distribution p θ (x|b).…”

Section: A Generative Modelmentioning

confidence: 90%

“…[3] showed that this fundamental difference between classic and variational autoencoders is relevant for hashing and yields to significantly better results. Later, [17] demonstrated that the use of Bernoulli instead of Gaussian latent variables helps to reduce the quantization loss arising from the use of continuous representations. This idea is also used in [4] and extended to incorporate supervision.…”

Section: Related Workmentioning

confidence: 99%

“…. , x (n) } denotes the set of training examples, the negative log-likelihood corresponding to a single data point x ( ) ∈ S, can be upper bounded by the following loss function [17]:…”

Section: Unsupervised Trainingmentioning

confidence: 99%

“…In the case of discrete distributions the gradients can be estimated using the so-called Gumbel-Softmax reparametrization trick [9]. Experiments in [17] show that this method is stable and effective for hashing.…”

Section: Unsupervised Trainingmentioning

confidence: 99%

See 3 more Smart Citations

Self-supervised Bernoulli Autoencoders for Semi-supervised Hashing

Ñanculef

Mena

Macaluso

et al. 2021

Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications

Self Cite

View full text Add to dashboard Cite

Semantic hashing is an emerging technique for large-scale similarity search based on representing highdimensional data using similarity-preserving binary codes used for efficient indexing and search. It has recently been shown that variational autoencoders, with Bernoulli latent representations parametrized by neural nets, can be successfully trained to learn such codes in supervised and unsupervised scenarios, improving on more traditional methods thanks to their ability to handle the binary constraints architecturally. However, the scenario where labels are scarce has not been studied yet.This paper investigates the robustness of hashing methods based on variational autoencoders to the lack of supervision, focusing on two semi-supervised approaches currently in use. The first augments the variational autoencoder's training objective to jointly model the distribution over the data and the class labels. The second approach exploits the annotations to define an additional pairwise loss that enforces consistency between the similarity in the code (Hamming) space and the similarity in the label space. Our experiments show that both methods can significantly increase the hash codes' quality. The pairwise approach can exhibit an advantage when the number of labelled points is large. However, we found that this method degrades quickly and loses its advantage when labelled samples decrease. To circumvent this problem, we propose a novel supervision method in which the model uses its label distribution predictions to implement the pairwise objective. Compared to the best baseline, this procedure yields similar performance in fully supervised settings but improves significantly the results when labelled data is scarce. Our code is made publicly available at https://github.com/amacaluso/SSB-VAE.

show abstract

Section: A Generative Modelmentioning

confidence: 90%

Section: Related Workmentioning

confidence: 99%

“…. , x (n) } denotes the set of training examples, the negative log-likelihood corresponding to a single data point x ( ) ∈ S, can be upper bounded by the following loss function [17]:…”

Section: Unsupervised Trainingmentioning

confidence: 99%

Section: Unsupervised Trainingmentioning

confidence: 99%

See 2 more Smart Citations

Self-supervised Bernoulli Autoencoders for Semi-supervised Hashing

Ñanculef

Mena

Macaluso

et al. 2021

Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications

Self Cite

View full text Add to dashboard Cite

show abstract

“…This reparametrization trick is applied by [25] to learn discrete latent variables. Another approach is based on variational inference [15]. One significant disadvantages of the above mentioned approaches is that the relational structure in the input space is only implicitly retained in latent space.…”

Section: Autoencodersmentioning

confidence: 99%

Semantic Preserving Siamese Autoencoder for Binary Quantization of Word Embeddings

Mostard

Schomaker

Wiering

2021

2021 5th International Conference on Natural Language Processing and Information Retrieval (NLPIR)

View full text Add to dashboard Cite

Word embeddings are used as building blocks for a wide range of natural language processing and information retrieval tasks. These embeddings are usually represented as continuous vectors, requiring significant memory capacity and computationally expensive similarity measures. In this study, we introduce a novel method for semantic hashing continuous vector representations into lowerdimensional Hamming space while explicitly preserving semantic information between words. This is achieved by introducing a Siamese autoencoder combined with a novel semantic preserving loss function. We show that our quantization model induces only a 4% loss of semantic information over continuous representations and outperforms the baseline models on several word similarity and sentence classification tasks. Finally, we show through cluster analysis that our method learns binary representations where individual bits hold interpretable semantic information. In conclusion, binary quantization of word embeddings significantly decreases time and space requirements while offering new possibilities through exploiting semantic information of individual bits in downstream information retrieval tasks.

show abstract

Logarithmic Continual Learning

et al. 2022

View full text Add to dashboard Cite

We introduce a neural network architecture that logarithmically reduces the number of selfrehearsal steps in the generative rehearsal of continually learned models. In continual learning (CL), training samples come in subsequent tasks, and the trained model can access only a current task. Contemporary CL methods employ generative models to replay previous samples and train them recursively with a combination of current and regenerated past data. This recurrence leads to superfluous computations as the same past samples are regenerated after each task, and the reconstruction quality successively degrades. In this work, we address these limitations and propose a new generative rehearsal architecture that requires, at most, a logarithmic number of retraining sessions for each sample. Our approach leverages the allocation of past data in a set of generative models such that most of them do not require retraining after a task. The experimental evaluation of our logarithmic continual learning approach shows the superiority of our method with respect to the state-of-the-art generative rehearsal methods.

show abstract

A Binary Variational Autoencoder for Hashing

Cited by 7 publications

References 10 publications

Self-supervised Bernoulli Autoencoders for Semi-supervised Hashing

Self-supervised Bernoulli Autoencoders for Semi-supervised Hashing

Semantic Preserving Siamese Autoencoder for Binary Quantization of Word Embeddings

Logarithmic Continual Learning

Contact Info

Product

Resources

About