Weighted Quantization-Regularization in DNNs for Weight Memory Minimization Toward HW Implementation

Wess, Matthias; Dinakarrao, Sai Manoj Pudukotai; Jantsch, Axel

doi:10.1109/tcad.2018.2857080

Cited by 31 publications

(6 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Exploding gradient problems: In addition to methods for solving the vanishing gradient problem, some other methods are exploited to tackle the exploding gradient problem, such as gradient clipping [38] and weight regularization [39]. There are two types of gradient clipping: the value clipping method is to clip the gradient that exceeds a preset threshold, and the norm clipping one adjusts the gradient according to its norm [38].…”

Section: Related Workmentioning

confidence: 99%

“…These regularization methods add a norm term to the loss function to softly constrain the parameter range. If the exploding gradient occurs (i.e., the norm of the parameter becomes very large), the regularization term can "pull back" the weight to a relatively flat region (i.e., the region which is close to zero), thus limit the occurrence of exploding gradients to some extent [39]. Nevertheless, the regularization term still remains unresolved issues on efficiency and stability.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Activated Gradients for Deep Neural Networks

Liu¹,

Chen²,

Du³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep neural networks often suffer from poor performance or even training failure due to the ill-conditioned problem, the vanishing/exploding gradient problem, and the saddle point problem. In this paper, a novel method by acting the gradient activation function (GAF) on the gradient is proposed to handle these challenges. Intuitively, the GAF enlarges the tiny gradients and restricts the large gradient. Theoretically, this paper gives conditions that the GAF needs to meet, and on this basis, proves that the GAF alleviates the problems mentioned above. In addition, this paper proves that the convergence rate of SGD with the GAF is faster than that without the GAF under some assumptions. Furthermore, experiments on CIFAR, ImageNet, and PASCAL visual object classes confirm the GAF's effectiveness. The experimental results also demonstrate that the proposed method is able to be adopted in various deep neural networks to improve their performance. The source code is publicly available at https://github.com/LongJin-lab/Activated-Gradients-for-Deep-Neural-Networks.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Activated Gradients for Deep Neural Networks

Liu¹,

Chen²,

Du³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Attempting to close the gap between the computational intensity of DNNs and the available computing power, a wide variety of hardware accelerators for DNNs and other AI workloads have emerged in recent years. A considerable amount of research has improved the efficiency of DNNs and reduced their memory consumption by applying methods such as pruning [5], [6], quantization [7]- [9], and factorization [10], [11]. Alternatively, a network architecture that is expected to work efficiently on the target device can be designed and trained directly.…”

Section: Introductionmentioning

confidence: 99%

ANNETTE: Accurate Neural Network Execution Time Estimation With Stacked Models

et al. 2021

Self Cite

View full text Add to dashboard Cite

“…In the algorithm side, light-weighted neural networks are used to optimize DNNs algorithm, which uses minimal neural layers and channels to complete the signal detection and classification [4]. In addition, the weight quantization and pruning optimization techniques have demonstrated superior performance [5] [6]. Both of them can improve the computation performance by reducing the scale of parameters hence to decrease computation operations and memory requirements.…”

Section: Introductionmentioning

confidence: 99%

Energy-Efficient Spin-Orbit Torque MRAM Operations for Neural Network Processor

Chang

Zhu

et al. 2021

2021 IEEE International Symposium on Circuits and Systems (ISCAS)

View full text Add to dashboard Cite

Emerging energy-efficient neural network processor is a promising hardware design to accelerate neural network algorithms with high performance and low power consumption. Typically, static random-access memory (SRAM) is employed to develop large buffers using in the processor. The bit cell of SRAM contains six transistors, leading to low density and large leakage current. In particular, several AI processors need multiple port and transfer-based SRAMs, which decrease the density and increase the power consumption. Recently, emerging spin-orbit torque magnetic random-access memory (SOT-MRAM) becomes a possible solution to replace the SRAM as working memory. However, more operations should be supported by the SOT-MRAM to provide sufficient functions, such as multiple-port memory, transpose memory, data-streaming operations. In this paper, we develop the working memory of neural network processor with SOT-MRAM to build the design library including the transpose operations, multiple-port memory, and data-streaming based buffer arrays. Equiped with those operations provided by SOT-MRAM, we can build high performance and energy-efficient neural network processors.

show abstract

Weighted Quantization-Regularization in DNNs for Weight Memory Minimization Toward HW Implementation

Cited by 31 publications

References 12 publications

Activated Gradients for Deep Neural Networks

Activated Gradients for Deep Neural Networks

ANNETTE: Accurate Neural Network Execution Time Estimation With Stacked Models

Energy-Efficient Spin-Orbit Torque MRAM Operations for Neural Network Processor

Contact Info

Product

Resources

About