Supporting Deep Neural Network Safety Analysis and Retraining Through Heatmap-Based Unsupervised Learning

Fahmy, Hazem; Pastore, Fabrizio; Bagherzadeh, Mojtaba; Briand, Lionel C.

doi:10.1109/tr.2021.3074750

Cited by 30 publications

(43 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Though this is to some extent subjective and context-dependent, we discuss whether SAFE finds a number of root cause clusters that is amenable to inspection by experts. To respond to this research question, like in the HUDD paper [8], we assume that experts visually inspect five images per root cause cluster to be able to make a decision. Under this assumption, we compare SAFE and HUDD in terms of the number of generated clusters and based on the ratio of error-inducing images that should be visually inspected when relying on each method.…”

Section: Resultsmentioning

confidence: 99%

“…To analyze the clusters, safety engineers can use the same approach as in HUDD [8]. For each cluster, a subset of elements is visually inspected to determine the associated unsafe conditions, as functional safety analysis requires.…”

Section: Functional Safety Analysis Through Root Cause Clusters Visua...mentioning

confidence: 99%

“…To retrain DNNs, we rely on the approach described in section 3.6. For fair comparisons with HUDD [8], BL1, and BL2, we configure bootstrap resampling to generate an augmented labeled unsafe set and an augmented labeled selected improvement set with the same size as the balanced labeled unsafe set for HUDD (see Figure 2).…”

Section: Rq13 Do Parameters With High Variance Reduction Represent a ...mentioning

confidence: 99%

“…SAFE makes use of transfer learning-based feature extraction, dimensionality reduction, and unsupervised learning. This approach is an improvement over HUDD [8] to avoid reliance on heatmap-based distance, which requires access to the DNN's internal information, and to improve the quality of the root cause clusters' identification. Transfer Learning transfers the knowledge from a generic domain to another specific domain using a pre-trained model.…”

Section: Introductionmentioning

confidence: 99%

“…When DNN-based systems are used in a safety-critical context, root cause analysis is required to support safety analysis [8]. A root cause is a source of a failure, which is in our context an incorrect DNN prediction or classification.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction and Clustering

Attaoui,

Fahmy,

Pastore

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

Deep neural networks (DNNs) have demonstrated superior performance over classical machine learning to support many features in safety-critical systems. Although DNNs are now widely used in such systems (e.g., self driving cars), there is limited progress regarding automated support for functional safety analysis in DNN-based systems. For example, the identification of root causes of errors, to enable both risk analysis and DNN retraining, remains an open problem. In this paper, we propose SAFE, a black-box approach to automatically characterize the root causes of DNN errors. SAFE relies on a transfer learning model pre-trained on ImageNet to extract the features from error-inducing images. It then applies a density-based clustering algorithm to detect arbitrary shaped clusters of images modeling plausible causes of error. Last, clusters are used to effectively retrain and improve the DNN. The black-box nature of SAFE is motivated by our objective not to require changes or even access to the DNN internals to facilitate adoption.Experimental results show the superior ability of SAFE in identifying different root causes of DNN errors based on case studies in the automotive domain. It also yields significant improvements in DNN accuracy after retraining, while saving significant execution time and memory when compared to alternatives. CCS Concepts: • Software and its engineering → Software defect analysis; • Computing methodologies → Machine learning.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Functional Safety Analysis Through Root Cause Clusters Visua...mentioning

confidence: 99%

Section: Rq13 Do Parameters With High Variance Reduction Represent a ...mentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction and Clustering

Attaoui,

Fahmy,

Pastore

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

To what extent do DNN-based image classification models make unreliable inferences?

Tian

Wen

et al. 2021

Empir Software Eng

View full text Add to dashboard Cite

Deep Neural Network (DNN) models are widely used for image classification. While they offer high performance in terms of accuracy, researchers are concerned about if these models inappropriately make inferences using features irrelevant to the target object in a given image. To address this concern, we propose a metamorphic testing approach that assesses if a given inference is made based on irrelevant features. Specifically, we propose two metamorphic relations (MRs) to detect such unreliable inferences. These relations expect (a) the classification results with different labels or the same labels but less certainty from models after corrupting the relevant features of images, and (b) the classification results with the same labels after corrupting irrelevant features. The inferences that violate the metamorphic relations are regarded as unreliable inferences.Our evaluation demonstrated that our approach can effectively identify unreliable inferences for single-label classification models with an average precision of 64.1% and 96.4% for the two MRs, respectively. As for multi-label classification models, the corresponding precision for MR-1 and MR-2 is 78.2% and 86.5%, respectively. Further, we conducted an empirical study to understand the problem of unreliable inferences in practice. Specifically, we applied our approach to 18 pre-trained single-label image classification models and 3 multi-label classification models, and then examined their inferences on the ImageNet and COCO datasets.We found that unreliable inferences are pervasive. Specifically, for each model, more than thousands of correct classifications are actually made using irrelevant features. Next, we investigated the effect of such pervasive unreliable inferences, and found that they can cause significant degradation of a model's overall accuracy.After including these unreliable inferences from the test set, the model's accuracy can be significantly changed. Therefore, we recommend that developers should pay more attention to these unreliable inferences during the model evaluations. We also explored the correlation between model accuracy and the size of unreliable inferences. We found the inferences of the input with smaller objects are easier to be unreliable. Lastly, we found that the current model training methodologies can guide the models to learn object-relevant features to certain extent, but may not necessarily prevent the model from making unreliable inferences. We encourage the community to propose more effective training methodologies to address this issue.

show abstract

Driver Posture Recognition: A Review

Mahomed,

Saha

2024

IEEE Access

View full text Add to dashboard Cite

Supporting Deep Neural Network Safety Analysis and Retraining Through Heatmap-Based Unsupervised Learning

Cited by 30 publications

References 40 publications

Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction and Clustering

Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction and Clustering

To what extent do DNN-based image classification models make unreliable inferences?

Driver Posture Recognition: A Review

Contact Info

Product

Resources

About