Deep Learning Training with Simulated Approximate Multipliers

Abstract

This paper presents by simulation how approximate multipliers can be utilized to enhance the training performance of convolutional neural networks (CNNs). Approximate multipliers have significantly better performance in terms of speed, power, and area compared to exact multipliers. However, approximate multipliers have an inaccuracy which is defined in terms of the Mean Relative Error (MRE). To assess the applicability of approximate multipliers in enhancing CNN training performance, a simulation for the impact of approximate multipliers error on CNN training is presented. The paper demonstrates that using approximate multipliers for CNN training can significantly enhance the performance in terms of speed, power, and area at the cost of a small negative impact on the achieved accuracy. Additionally, the paper proposes a hybrid training method which mitigates this negative impact on the accuracy. Using the proposed hybrid method, the training can start using approximate multipliers then switches to exact multipliers for the last few epochs. Using this method, the performance benefits of approximate multipliers in terms of speed, power, and area can be attained for a large portion of the training stage. On the other hand, the negative impact on the accuracy is diminished by using the exact multipliers for the last epochs of training.

A Survey on Approximate Multiplier Designs for Energy Efficiency: From Algorithms to Circuits

“…Such circuit-level techniques can be integrated into the designs at the other two levels. Approximate multipliers have already been successfully applied in error-tolerant tasks, such as filtering, image processing, and machine learning [14,15,[37][38][39][59][60][61][62][63][64][65][66][67][68][69][70][71][72]. In 1985, Ashtaputre et al designed a systolic array with approximate multipliers and demonstrated that the inaccurate multiplication can bring negligible impact on the results [59].…”

“…Hammad et al replaced the original full-precision multipliers in VGG networks with approximate ones to support the classification tasks on CIFAR-10 and CIFAR-100, which again showed negligible accuracy losses [68]. After that, they further proposed to deploy approximate multipliers in training to improve the performance [70].…”

CNN Inference Using a Preprocessing Precision Controller and Approximate Multipliers With Various Precisions

“…The utilization of approximate multipliers in the hardware design of convolutional neural networks (CNNs) has been proposed previously to enhance the performance in terms of power, speed, and area [23][24][25][26][27][28]. Moreover, using a reconfigurable approximate multiplier based on calculating the error variance was proposed in [29].…”

“…Moreover, using a reconfigurable approximate multiplier based on calculating the error variance was proposed in [29]. Lower precision approximate multipliers can achieve higher performance gains as can be seen in [23][24][25][26][27]. However, this performance enhancement has a cost of a drop in the CNN accuracy which is inversely proportional to the precision.…”

Dual segmentation approximate multiplier

“…Introduction: Approximate computing can be applied in error-resilient applications to reduce power, area, and delay [1][2][3][4][5][6][7][8][9][10]. Multiplication is a fundamental high-energy operation in image processing and deep learning applications [11][12][13][14]. Prior works have explored different techniques to reduce the cost of multiplication using approximate multipliers.…”