Variational Deep Semantic Hashing for Text Documents

Abstract

As the amount of textual data has been rapidly increasing over the past decade, efficient similarity search methods have become a crucial component of large-scale information retrieval systems. A popular strategy is to represent original data samples by compact binary codes through hashing. A spectrum of machine learning methods have been utilized, but they o en lack expressiveness and flexibility in modeling to learn effective representations. e recent advances of deep learning in a wide range of applications has demonstrated its capability to learn robust and powerful feature representations for complex data. Especially, deep generative models naturally combine the expressiveness of probabilistic generative models with the high capacity of deep neural networks, which is very suitable for text modeling. However, li le work has leveraged the recent progress in deep learning for text hashing.In this paper, we propose a series of novel deep document generative models for text hashing. e first proposed model is unsupervised while the second one is supervised by utilizing document labels/tags for hashing. e third model further considers document-specific factors that affect the generation of words. e probabilistic generative formulation of the proposed models provides a principled framework for model extension, uncertainty estimation, simulation, and interpretability. Based on variational inference and reparameterization, the proposed models can be interpreted as encoder-decoder deep neural networks and thus they are capable of learning complex nonlinear distributed representations of the original documents. We conduct a comprehensive set of experiments on four public testbeds. e experimental results have demonstrated the effectiveness of the proposed supervised learning models for text hashing.

Document Hashing with Mixture-Prior Generative Models

“…This phenomenon is especially obvious for VDSH, in which the precisions on all three datasets drop by a significant margin. This interesting phenomenon has been reported in previous works Chaidaroon and Fang, 2017;Wang et al, 2013;Liu et al, 2012), and the reason could be overfitting since the model with long hashing codes is more likely to overfitting (Chaidaroon and Fang, 2017;. However, it can be seen that our model is more robust to the number of hashing bits.…”

“…Let x ∈ Z |V | + denote the bag-of-words representation of a document and x i ∈ {0, 1} |V | denote the one-hot vector representation of the i-th word of the document, where |V | denotes the vocabulary size. VDSH in (Chaidaroon and Fang, 2017) proposed to model a document D, which is de-fined by a sequence of one-hot word representa-…”

“…Specifically, a two-layer feed-forward neural network with 500 hidden units and ReLU activation function is employed as the encoder and the extra classifier in the supervised case, while the decoder is the same as that stated in (2). Similar to VDSH and NASH (Chaidaroon and Fang, 2017;, the TFIDF feature of a document is used as the input to the encoder. The Adam optimizer (Kingma and Ba, 2014) is used in Datasets TMC 20Newsgroups Reuters Method 16bit 32bit 64bit 128bit 16bit 32bit 64bit 128bit 16bit 32bit 64bit the training of our models, and its learning rate is set to be 1 × 10 −3 , with a decay rate of 0.96 for every 10000 iterations.…”

“…Many efforts have been devoted to hashing by using generative models. In (Chaidaroon and Fang, 2017), variational deep semantic hashing (VDSH) is proposed to solve the semantic hashing problem by using the variational autoencoder (VAE) (Kingma and Welling, 2013). However, this model requires a two-stage training since a separate step is needed to cast the continuous representations in VAE into binary codes.…”

Unsupervised Neural Generative Semantic Hashing

“…This can be done using techniques similar to Latent Semantic Indexing [33], spectral clustering [29], or two-step approaches of first creating an optimal encoding and then training a classifier to predict this [34]. Recent work has focused on deep learning based methods [4,5,23] to create a generative document model. However, none of the methods directly model the end goal of providing an effective similarity search, i.e., being able to accurately rank documents based on their hash codes, but rather just focus solely on generating document representations.…”