Improving transparency of deep neural inference process

Abstract

Deep learning techniques are rapidly advanced recently, and becoming a necessity component for widespread systems. However, the inference process of deep learning is black-box, and not very suitable to safety-critical systems which must exhibit high transparency. In this paper, to address this black-box limitation, we develop a simple analysis method which consists of 1) structural feature analysis: lists of the features contributing to inference process, 2) linguistic feature analysis: lists of the natural language labels describing the visual attributes for each feature contributing to inference process, and 3) consistency analysis: measuring consistency among input data, inference (label), and the result of our structural and linguistic feature analysis. Our analysis is simplified to reflect the actual inference process for high transparency, whereas it does not include any additional black-box mechanisms such as LSTM for highly human readable results. We conduct experiments and discuss the results of our analysis qualitatively and quantitatively, and come to believe that our work improves the transparency of neural networks. Evaluated through 12,800 human tasks, 75% workers answer that input data and result of our feature analysis are consistent, and 70% workers answer that inference (label) and result of our feature analysis are consistent. In addition to the evaluation of the pro-

A survey on matching strategies for boundary image comparison and evaluation

“…Secondly, if many differently trained CNNs yield dissimilar results, how to produce a canonical comparer out of them? Alternatively, if using any of such CNNs for the task, how to audit their results [66] ? Thirdly, as reported by Nguyen et al [67] , wellperforming CNNs can lead to aberrant results that are misleading, even if they are produced with an almost-full ( > 99% ) confidence.…”

Engineering problems in machine learning systems

“…Further, interpretability is also useful for performance improvement, debugging during training, and validating of training results. Developers can understand the internal behavior of a trained NN to train higher performance models [45]. For example, a developer can visualize an NN's focus points for an incorrect inference and understand what was wrong, before additional training data is collected according to the analysis.…”

Adapting SQuaRE for Quality Assessment of Artificial Intelligence Systems

“…A quality sub-characteristic Operability has a measure Monitoring capability. Explainable AI (XAI) [8] is a rapid growing area in the artificial intelligence research, and techniques to explain or interpret ML components are proposed in recent years [13], [14]. They can be used for monitoring capacities for ML-based AI systems, but XAI research is still in the very early stage.…”