Special Session: Fault-Tolerant Deep Learning: A Hierarchical Perspective

Liu, Cheng; Gao, Zhen; Liu, Siting; Ning, Xuefei; Li, Huawei; Li, Xiaowei

doi:10.1109/vts52500.2021.9794239

Cited by 8 publications

(4 citation statements)

References 127 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It covered resilience threats related to transient, permanent, and TEs induced by process variation, memory refresh rate scaling, voltage scaling, and thermal effects. The study of [11] proposes a survey on fault-tolerant deep learning with a hierarchical approach studying proposed methods from the model layer, architecture layer, circuit layer, and cross-layer views. Moreover, [12] reviews the reliability issues of DNN accelerators considering soft errors.…”

Section: A Papers With Similar Backgroundmentioning

confidence: 99%

Dependable DNN Accelerator for Safety-Critical Systems: A Review on the Aging Perspective

Moghaddasi,

Gorgin,

Lee

2023

IEEE Access

View full text Add to dashboard Cite

Nowadays, artificial intelligence (AI) and deep learning (DL) progressively adapt to various spheres of our lives. These disciplines contain safety-critical applications such as autonomous driving with a high risk of human injury in the case of malfunction, requiring a high promise of dependability. Even the dependability becomes more crucial as shrinking CMOS technology feature size worsens the resilience concerns due to factors like aging. This paper addresses the overarching dependability issue of advanced deep neural networks (DNN) accelerators from the aging perspective. Especially, a comprehensive survey and taxonomy of techniques used to evaluate and mitigate aging effects are introduced. We cover different aging effects like permanent faults, timing errors, and lifetime issues. We review research by the layer-wise approach and categorize several resilience classes to bring out major features. The concluding part of this review highlights the questions answered and several future research directions. This study is expected to benefit researchers in different areas of DNN deployment, especially the dependability of this emergent paradigm.

show abstract

Section: A Papers With Similar Backgroundmentioning

confidence: 99%

Dependable DNN Accelerator for Safety-Critical Systems: A Review on the Aging Perspective

Moghaddasi,

Gorgin,

Lee

2023

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Some of the reliability evaluations explored the reliability difference among components of neural networks such that they can be utilized to perform selective protection with less protection overhead [43]- [45]. More neural network reliability evaluation works can be found in recent surveys [46] [47].…”

Section: B Reliability Analysis Of Neural Networkmentioning

confidence: 99%

Exploring Winograd Convolution for Cost-Effective Neural Network Fault Tolerance

Xue,

Liu,

Liu

et al. 2023

IEEE Trans. VLSI Syst.

View full text Add to dashboard Cite

Winograd is generally utilized to optimize convolution performance and computational efficiency because of the reduced multiplication operations, but the reliability issues brought by winograd are usually overlooked. In this work, we observe the great potential of winograd convolution in improving neural network (NN) fault tolerance. Based on the observation, we evaluate winograd convolution fault tolerance comprehensively from different granularities ranging from models, layers, and operation types for the first time. Then, we explore the use of inherent fault tolerance of winograd convolution for cost-effective NN protection against soft errors. Specifically, we mainly investigate how winograd convolution can be effectively incorporated with classical fault-tolerant design approaches including triple modular redundancy (TMR), faultaware retraining, and constrained activation functions. According to our experiments, winograd convolution can reduce the faulttolerant design overhead by 55.77% on average without any accuracy loss compared to standard convolution, and further reduce the computing overhead by 17.24% when the inherent fault tolerance of winograd convolution is considered. When it is applied on fault-tolerant neural networks enhanced with faultaware retraining and constrained activation functions, the resulting model accuracy generally shows significant improvement in presence of various faults.

show abstract

“…Fault simulation is key to understand the influence of hardware faults on the neural network processing and is the basis for fault-tolerant neural network model and accelerator designs [18] [12] [19] [4] [10] in various application scenarios. For instance, fault simulations in [36] [33] are utilized to investigate the vulnerability of neural networks and accelerators, which enables selectively hardware protection against various hardware faults with minimum overhead.…”

Section: Related Workmentioning

confidence: 99%

“…The reliability of neural network accelerators that are increasingly utilized for their competitive advantages in terms of performance and energy efficiency [30] [29] becomes critical to these applications, and must be eval- With the continuously shrinking semiconductor feature sizes and growing transistor density, the influence of soft errors on large-scale chip designs becomes inevitable [5] [31]. A variety of analysis work have been conducted to investigate the influence of soft errors on neural network execution reliability from distinct angles recently [27] [14] [34] [35] [36] [32] [13] [20] [6] [12] [18]. For instance, Brandon Reagen et al [27] investigated the relationship between fault error rate and model accuracy from the perspective of models, layers, and structures.…”

Section: Introductionmentioning

confidence: 99%

Statistical Modeling of Soft Error Influence on Neural Networks

Huang¹,

Xue²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Soft errors in large VLSI circuits pose dramatic influence on computing-and memory-intensive neural network (NN) processing. Understanding the influence of soft errors on NNs is critical to protect against soft errors for reliable NN processing. Prior work mainly rely on fault simulation to analyze the influence of soft errors on NN processing. They are accurate but usually specific to limited configurations of errors and NN models due to the prohibitively slow simulation speed especially for large NN models and datasets. With the observation that the influence of soft errors propagates across a large number of neurons and accumulates as well, we propose to characterize the soft error induced data disturbance on each neuron with normal distribution model according to central limit theorem and develop a series of statistical models to analyze the behavior of NN models under soft errors in general. The statistical models reveal not only the correlation between soft errors and NN model accuracy, but also how NN parameters such as quantization and architecture affect the reliability of NNs. The proposed models are compared with fault simulation and verified comprehensively. In addition, we observe that the statistical models that characterize the soft error influence can also be utilized to predict fault simulation results in many cases and we explore the use of the proposed statistical models to accelerate fault simulations of NNs. According to our experiments, the accelerated fault simulation shows almost two orders of magnitude speedup with negligible simulation accuracy loss over the baseline fault simulations.

show abstract

Special Session: Fault-Tolerant Deep Learning: A Hierarchical Perspective

Cited by 8 publications

References 127 publications

Dependable DNN Accelerator for Safety-Critical Systems: A Review on the Aging Perspective

Dependable DNN Accelerator for Safety-Critical Systems: A Review on the Aging Perspective

Exploring Winograd Convolution for Cost-Effective Neural Network Fault Tolerance

Statistical Modeling of Soft Error Influence on Neural Networks

Contact Info

Product

Resources

About