Quickly Generating Representative Samples from an RBM-Derived Process

Abstract

Two recently proposed learning algorithms, herding and fast persistent contrastive divergence (FPCD), share the following interesting characteristic: they exploit changes in the model parameters while sampling in order to escape modes and mix better during the sampling process that is part of the learning algorithm. We justify such approaches as ways to escape modes while keeping approximately the same asymptotic distribution of the Markov chain. In that spirit, we extend FPCD using an idea borrowed from Herding in order to obtain a pure sampling algorithm, which we call the rates-FPCD sampler. Interestingly, this sampler can improve the model as we collect more samples, since it optimizes a lower bound on the log likelihood of the training data. We provide empirical evidence that this new algorithm displays substantially better and more robust mixing than Gibbs sampling.

The role of different sampling methods in improving biological activity prediction using deep belief network

“…The main application of this network is to recognize the best values for initializing the learning procedure to fine‐tune the weights and biases of ANN. The commonly used method in constructing each layer of DBN is restricted Boltzmann machine (RBM) …”

“…This iterative approach is a randomizing algorithm and guarantees obtaining the best results from the model. It is obvious that running this algorithm for many steps is too time consuming to be practical . Hinton et al (2006) introduced a fast greedy algorithm that quickly produces a fairly good set of parameters, even in deep networks with millions of parameters and many hidden layers using contrastive divergence (CD) method .…”

“…In 2009, PCD was improved by Teileman and Hinton to provide fast persistent contrastive divergence (FPCD) algorithm obtained by decoupling the parameters used in positive and negative phases . In this model, adding the new parameter, “fast,” to the update step caused the weights and biases to learn rapidly and caused the model to be optimized to reach the global minimum …”

“…The commonly used method in constructing each layer of DBN is restricted Boltzmann machine (RBM). [11] RBM is an energy-based method used as a discriminative or generative model for labeled or unlabeled data. It has a single layer of hidden units with no internal layer of visible and hidden neurons.…”

Asynchronous network of cellular automaton-based neurons for efficient implementation of Boltzmann machines

“…They demonstrate that the Boltzmann machine implemented using the ANCAN generated a wide variety of images compared to the other Boltzmann machines. Following previous works on generative models [29][30][31], we evaluated the generation abilities of the Boltzmann machines by the log-likelihood of the test subset. The probability of the test subset was estimated by fitting a Gaussian Parzen window to the samples from the Boltzmann machines.…”