A Novel Restricted Boltzmann Machine Training Algorithm with Fast Gibbs Sampling Policy

Abstract

The restricted Boltzmann machine (RBM) is one of the widely used basic models in the field of deep learning. Although many indexes are available for evaluating the advantages of RBM training algorithms, the classification accuracy is the most convincing index that can most effectively reflect its advantages. RBM training algorithms are sampling algorithms essentially based on Gibbs sampling. Studies focused on algorithmic improvements have mainly faced challenges in improving the classification accuracy of the RBM training algorithms. To address the above problem, in this paper, we propose a fast Gibbs sampling (FGS) algorithm to learn the RBM by adding accelerated weights and adjustment coefficient. An important link based on Gibbs sampling theory was established between the update of the network weights and mixing rate of Gibbs sampling chain. The proposed FGS method was used to accelerate the mixing rate of Gibbs sampling chain by adding accelerated weights and adjustment coefficients. To further validate the FGS method, numerous experiments were performed to facilitate comparisons with the classical RBM algorithm. The experiments involved learning the RBM based on standard data. The results showed that the proposed FGS method outperformed the CD, PCD, PT5, PT10, and DGS algorithms, particularly with respect to the handwriting database. The findings of our study suggest the potential applications of FGS to real-world problems and demonstrate that the proposed method can build an improved RBM for classification.

A Scalable System-on-Chip Acceleration for Deep Neural Networks

“…An RBM is a special class of Boltzmann machines (BM). The BM is a parallel computational model for the implementation of simulated annealing which is one of the frequently used heuristic search algorithms [26], [27]. It is a stochastic neural network which can learn internal representations to solve combinatorial optimization problems.…”

“…Practically speaking, an RBM is employed in several applications due to a relatively simpler training process as compared to an UBM architecture. Conventional training algorithms for RBMs are not efficient in terms of time due to their slow convergence rate [26]. Therefore, RBMs are generally trained with approximate training algorithms [27].…”

Sensor Fault Diagnosis Based on Adaptive Arc Fuzzy DBN-Petri Net

“…The FGS algorithm introduces the accelerated confidence-weight f astW and the adjustment coefficient ξ, and the confidence-weights of the entire network are the sum of the acceleration confidenceweights f astW and the traditional confidence-weights, effectively ensuring that the confidence-weights updates are updated rapidly in the early stages of training. The added adjustment coefficient can effectively change the update rate of the accelerated confidence-weights, and the adjustment coefficient changes within the range of zero to one, effectively reducing the trend of accelerated confidence-weights update in the middle and later stages of training [29].…”

“…DBN is a generated graphical model that can be regarded as a deep neural network. Restricted Boltzmann machine (RBM) is a widely used Markov random field (MRF) model, and it is also an important model that constitutes DBN [29]. This model is based on the energy function and can express random neural networks.…”

“…Gibbs sampling is a well-known Markov Monte Carlo (MCMC) algorithm that can extract samples from complex high-dimensional posterior distributions and is often used in randomization algorithms for Bayesian inference [31]. The fast Gibbs sampling (FGS) [29] algorithm based on Gibbs sampling adds acceleration weights and adjustment coefficients, significantly improving the mixing speed of the sampling chain and the accuracy of RBM training. Most of the existing improved FGS algorithms achieve a faster convergence rate at the data and algorithm level to achieve better prediction results [32]- [36].…”