A 41.3/26.7 pJ per Neuron Weight RBM Processor Supporting On-Chip Learning/Inference for IoT Applications

Tsai, Chang-Hung; Wang, Yu; Wong, Wing Hung; Lee, Chen-Yi

doi:10.1109/jssc.2017.2715171

Cited by 18 publications

(6 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In fact, binary stochastic neurons are desired for deep learning networks, but are typically avoided because it is harder to generate random bits in CMOS hardware 77 . Use of this compact neuron that relies on MTJs natural physics to provide stochastic binarization could accelerate computation in custom hardware 78,79 by faster evaluation of BSN function 32 and also encourage algorithmic advancement using BSN.…”

Section: Discussionmentioning

confidence: 99%

Quantitative Evaluation of Hardware Binary Stochastic Neurons

Hassan

Datta

2021

Phys. Rev. Applied

View full text Add to dashboard Cite

Recently there has been increasing activity to build dedicated Ising Machines to accelerate the solution of combinatorial optimization problems by expressing these problems as a ground-state search of the Ising model. A common theme of such Ising Machines is to tailor the physics of underlying hardware to the mathematics of the Ising model to improve some aspect of performance that is measured in speed to solution, energy consumption per solution or area footprint of the adopted hardware. One such approach to build an Ising spin, or a binary stochastic neuron (BSN), is a compact mixed-signal unit based on a low-barrier nanomagnet based design that uses a single magnetic tunnel junction (MTJ) and three transistors (3T-1MTJ) where the MTJ functions as a stochastic resistor (1SR). Such a compact unit can drastically reduce the area footprint of BSNs while promising massive scalability by leveraging the existing Magnetic RAM (MRAM) technology that has integrated 1T-1MTJ cells in ∼ Gbit densities. The 3T-1SR design however can be realized using different materials or devices that provide naturally fluctuating resistances. Extending previous work, we evaluate hardware BSNs from this general perspective by classifying necessary and sufficient conditions to design a fast and energy-efficient BSN that can be used in scaled Ising Machine implementations. We connect our device analysis to systems-level metrics by emphasizing hardware-independent figures-of-merit such as flips per second and dissipated energy per random bit that can be used to classify any Ising Machine.

show abstract

Section: Discussionmentioning

confidence: 99%

Quantitative Evaluation of Hardware Binary Stochastic Neurons

Hassan

Datta

2021

Phys. Rev. Applied

View full text Add to dashboard Cite

show abstract

“…Thus, it seems as [38] is optimized for 16 bit× 16 bit. In general, higher flexibility comes with the price of reduced efficiency [38], [41], [42]. For this reason, most hardware accelerators for DNNs support only a limited number of DNNs very efficiently.…”

Section: ) Quantizationmentioning

confidence: 99%

“…Content may change prior to final publication. [25], [30], [32]- [38], [42], [46], [47], [53], [55], [57], [ [29], [31], [39], [40], [45] → → → ↑ Compression I [49], [51], [53], [55], [57], [ [25], [36]- [38], [48], [49], [51], [57] ↓ ↑ ↓ → Zero-Skipping I [49], [51], [53], [55], [57], [58] ↓ ↑ ↓ → Bit-Serial Processing I [24], [25], [34], [36], [37]…”

Section: ) Voltage Underscalingmentioning

confidence: 99%

Efficiency Versus Accuracy: A Review of Design Techniques for DNN Hardware Accelerators

Latotzke

Gemmeke

2021

IEEE Access

View full text Add to dashboard Cite

Deep neural networks (DNNs) have surpassed other algorithms in analyzing today's abundant data. Due to the security and delay requirements of the given applications, analytical data extraction should happen on edge devices. Edge devices, however, struggle to support the increasingly complex DNNs because of the models' high computational load and number of employed parameters. Thus, edge devices need support by efficient accelerators to process DNNs. However, the design of DNN accelerators remains challenging, as there is a lack of established design techniques directed towards a specific design point in terms of energy budget, area, time-to-solution, and classification accuracy. This paper fills this gap by providing a quantitative, large-scale, state-of-the-art comparison of DNN accelerators building on published data subjected to technology scaling and benchmark normalization. The leveled comparison stimulates learning from previous designs by considering the impact of each technique on energy, area, and timeto-solution. Furthermore, the key design techniques used in the DNN accelerators are classified according to their influence on the classification accuracy. Finally, we provide a discussion of hardware accelerators to support future designers in considering the trade-off between efficiency and accuracy in order to identify the most suitable techniques for certain benchmarks.

show abstract

“…e RBM is a stochastic generative neural network that can learn the probability distribution from the input datasets. RBMs are applied in dimension reduction [7], classification [8], collaborative filtering [9], feature learning [10], topic modeling [11], radar target automatic recognition [12], chip synthesis [13], and speech recognition [14]. RBMs can be trained by either supervised or unsupervised learning depending on the different tasks.…”

Section: Introductionmentioning

confidence: 99%

A Novel Restricted Boltzmann Machine Training Algorithm with Fast Gibbs Sampling Policy

Wang

Gao

Wan

et al. 2020

Mathematical Problems in Engineering

View full text Add to dashboard Cite

The restricted Boltzmann machine (RBM) is one of the widely used basic models in the field of deep learning. Although many indexes are available for evaluating the advantages of RBM training algorithms, the classification accuracy is the most convincing index that can most effectively reflect its advantages. RBM training algorithms are sampling algorithms essentially based on Gibbs sampling. Studies focused on algorithmic improvements have mainly faced challenges in improving the classification accuracy of the RBM training algorithms. To address the above problem, in this paper, we propose a fast Gibbs sampling (FGS) algorithm to learn the RBM by adding accelerated weights and adjustment coefficient. An important link based on Gibbs sampling theory was established between the update of the network weights and mixing rate of Gibbs sampling chain. The proposed FGS method was used to accelerate the mixing rate of Gibbs sampling chain by adding accelerated weights and adjustment coefficients. To further validate the FGS method, numerous experiments were performed to facilitate comparisons with the classical RBM algorithm. The experiments involved learning the RBM based on standard data. The results showed that the proposed FGS method outperformed the CD, PCD, PT5, PT10, and DGS algorithms, particularly with respect to the handwriting database. The findings of our study suggest the potential applications of FGS to real-world problems and demonstrate that the proposed method can build an improved RBM for classification.

show abstract

A 41.3/26.7 pJ per Neuron Weight RBM Processor Supporting On-Chip Learning/Inference for IoT Applications

Cited by 18 publications

References 10 publications

Quantitative Evaluation of Hardware Binary Stochastic Neurons

Quantitative Evaluation of Hardware Binary Stochastic Neurons

Efficiency Versus Accuracy: A Review of Design Techniques for DNN Hardware Accelerators

A Novel Restricted Boltzmann Machine Training Algorithm with Fast Gibbs Sampling Policy

Contact Info

Product

Resources

About