ERDNN: Error-Resilient Deep Neural Networks With a New Error Correction Layer and Piece-Wise Rectified Linear Unit

Abstract

Deep Learning techniques have been successfully used to solve a wide range of computer vision problems. Due to their high computation complexity, specialized hardware accelerators are being proposed to achieve high performance and efficiency for deep learning-based algorithms. However, soft errors, i.e., bit flipping errors in the layer output, are often caused due to process variation and high energy particles in these hardware systems. These can significantly reduce model accuracy. To remedy this problem, we propose new algorithms that effectively reduce the impact of errors, thus keeping high accuracy. We firstly propose to incorporate an Error Correction Layer (ECL) into neural networks where convolution is performed multiple times in each layer and majority reporting is conducted for the outputs at bit level. We found that ECL can eliminate most errors while bypassing the bit-error when the bits at the same position are corrupted multiple times under the simulated condition. In order to solve this problem, we analyze the impact of errors depending on the position of bits, thus observing that errors in most significant bit (MSB) positions tend to severely corrupt the output of the network compared to the errors in the least significant bit (LSB) positions. According to this observation, we propose a new specialized activation function, called Piece-wise Rectified Linear Unit (PwReLU), which selectively suppresses errors depending on the bit positions, resulting in an increased model resistance against the errors. Compared to existing activation functions, the proposed PwReLU outperforms with large accuracy margins of up-to 20% even with very high bit error rates (BERs). Our extensive experiments show that the proposed ECL and PwReLU work in a complementary manner, achieving comparable accuracy to the error-free networks even at a severe BER of 0.1% on CIFAR10, CIFAR100, and ImageNet.

A Super-Resolution-Based Feature Map Compression for Machine-Oriented Video Coding

“…According to a recent collaborative intelligence study [14], it is possible to efficiently spread compute burdens across cloud and edge devices and reduce energy usage in most circumstances. As mobile devices grow more capable and energy-efficient, performing feature extraction calculations on the front-end mobile device can provide computational power offloading while decreasing energy usage in the back-end data center [11]. Bajic et al [15] claim that transmitting the features can also prevent privacy issues due to the complexity of feature interpretation by an intercepter.…”

Binary-Classifiers-Enabled Filters for Semi-Supervised Learning

“…Deep learning based schemes have witnessed a significant performance improvement for a wide variety of emerging applications ranging from wireless systems to computer vision [1], [2]. These applications include image classification [3]- [5], text classification [5]- [8], audio classification [9]- [11], object detection [12], stock market analysis [13], smart city [13] and many more [12], [13].…”