Ares: A framework for quantifying the resilience of deep neural networks

Reagen, Brandon; Gupta, Udit; Pentecost, Lillian; Whatmough, Paul N.; Lee, Sae Kyu; Mulholland, Niamh; Brooks, D.; Wei, Gu-Yeon

doi:10.1109/dac.2018.8465834

Cited by 152 publications

(121 citation statements)

References 3 publications

Supporting

Mentioning

114

Contrasting

Order By: Relevance

“…The degree of noise or fault tolerance can vary significantly across different neural network models [33], but interestingly, such models can be made robust via proper construction and training [34]- [37]. In some cases, unbiased noise added during training results in a more robust model, effectively acting as a form of regularization [38].…”

Section: B Precisionmentioning

confidence: 99%

Photonic Multiply-Accumulate Operations for Neural Networks

Nahmias

Lima

Tait

et al. 2020

IEEE J. Select. Topics Quantum Electron.

237

133

View full text Add to dashboard Cite

It has long been known that photonic communication can alleviate the data movement bottlenecks that plague conventional microelectronic processors. More recently, there has also been interest in its capabilities to implement low precision linear operations, such as matrix multiplications, fast and efficiently. We characterize the performance of photonic and electronic hardware underlying neural network models using multiply-accumulate operations. First, we investigate the limits of analog electronic crossbar arrays and on-chip photonic linear computing systems. Photonic processors are shown to have advantages in the limit of large processor sizes (>100 µm), large vector sizes (N > 500), and low noise precision (≤4 bits). We discuss several proposed tunable photonic MAC systems, and provide a concrete comparison between deep learning and photonic hardware using several empiricallyvalidated device and system models. We show significant potential improvements over digital electronics in energy (>10 2), speed (>10 3), and compute density (>10 2). Index Terms-Artificial intelligence, neural networks, analog computers, analog processing circuits, optical computing. I. INTRODUCTION P HOTONICS has been well studied for its role in communication systems. Fiber optic links currently form the backbone of the world's telecommunications infrastructure, vastly overshadowing the best electronic communication standards in information capacity. Light signals have many advantageous properties for the transfer of information. For one, a photonic waveguide, with diameters ranging from those in fiber (∼80 μm) to those fabricated on-chip (∼500 nm), can carry information at enormous bandwidth densities-i.e., terabits per secondwith an energy efficiency that scales nearly independent of distance. This density is possible thanks to signal parallelization in photonic waveguides, in which hundreds of high speed, multiplexed channels can be independently modulated and detected. Photonic channels also experience less distortion, jitter,

show abstract

Section: B Precisionmentioning

confidence: 99%

Photonic Multiply-Accumulate Operations for Neural Networks

Nahmias

Lima

Tait

et al. 2020

IEEE J. Select. Topics Quantum Electron.

237

133

View full text Add to dashboard Cite

show abstract

“…8bit fixed point representation was used through the experiments. The error injection is measured with bit error rate (BER) according to [9]. In addition, we also had random errors injected to the internal computing results.…”

Section: Methodsmentioning

confidence: 99%

Resilient Neural Network Training for Accelerators with Computing Errors

Xing

Liu

et al. 2019

2019 IEEE 30th International Conference on Application-Specific Systems, Architectures and Processors (ASAP)

View full text Add to dashboard Cite

With the advancements of neural networks, customized accelerators are increasingly adopted in massive AI applications. To gain higher energy efficiency or performance, many hardware design optimizations such as near-threshold logic or overclocking can be utilized. In these cases, computing errors may happen and the computing errors are difficult to be captured by conventional training on general purposed processors (GPPs). Applying the offline trained neural network models to the accelerators with errors directly may lead to considerable prediction accuracy loss. To address this problem, we explore the resilience of neural network models and relax the accelerator design constraints to enable aggressive design options. First of all, we propose to train the neural network models using the accelerators' forward computing results such that the models can learn both the data and the computing errors. In addition, we observe that some of the neural network layers are more sensitive to the computing errors. With this observation, we schedule the most sensitive layer to the attached GPP to reduce the negative influence of the computing errors. According to the experiments, the neural network models obtained from the proposed training outperform the original models significantly when the CNN accelerators are affected by computing errors.

show abstract

“…Applications like image classification, which generate only one output (i.e., class of the image) per input sample are considered to be more error-resilient as compared to the applications like object detection, which produce more sophisticated output. Various techniques based on fault/noise injection have been proposed to evaluate the errorresilience of the DNNs [55] [18]. These techniques help in quantifying the amount of approximation that can be applied in a DNN.…”

Section: Hardware-level Optimizationsmentioning

confidence: 99%

Deep Learning for Edge Computing: Current Trends, Cross-Layer Optimizations, and Open Research Challenges

Marchisio

Hanif

Khalid

et al. 2019

2019 IEEE Computer Society Annual Symposium on VLSI (ISVLSI)

View full text Add to dashboard Cite

In the Machine Learning era, Deep Neural Networks (DNNs) have taken the spotlight, due to their unmatchable performance in several applications, such as image processing, computer vision, and natural language processing. However, as DNNs grow in their complexity, their associated energy consumption becomes a challenging problem. Such challenge heightens for edge computing, where the computing devices are resource-constrained while operating on limited energy budget. Therefore, specialized optimizations for deep learning have to be performed at both software and hardware levels. In this paper, we comprehensively survey the current trends of such optimizations and discuss key open research mid-term and long-term challenges.

show abstract

Ares: A framework for quantifying the resilience of deep neural networks

Cited by 152 publications

References 3 publications

Photonic Multiply-Accumulate Operations for Neural Networks

Photonic Multiply-Accumulate Operations for Neural Networks

Resilient Neural Network Training for Accelerators with Computing Errors

Deep Learning for Edge Computing: Current Trends, Cross-Layer Optimizations, and Open Research Challenges

Contact Info

Product

Resources

About