Is Deep Learning Safe for Robot Vision? Adversarial Examples Against the iCub Humanoid

Abstract

Deep neural networks have been widely adopted in recent years, exhibiting impressive performances in several application domains. It has however been shown that they can be fooled by adversarial examples, i.e., images altered by a barely-perceivable adversarial noise, carefully crafted to mislead classification. In this work, we aim to evaluate the extent to which robot-vision systems embodying deeplearning algorithms are vulnerable to adversarial examples, and propose a computationally efficient countermeasure to mitigate this threat, based on rejecting classification of anomalous inputs. We then provide a clearer understanding of the safety properties of deep networks through an intuitive empirical analysis, showing that the mapping learned by such networks essentially violates the smoothness assumption of learning algorithms. We finally discuss the main limitations of this work, including the creation of real-world adversarial examples, and sketch promising research directions. 1

Optimization and abstraction: a synergistic approach for analyzing neural network robustness

“…Szegedy et al [51] first showed that neural networks are vulnerable to small perturbations on inputs. It has since been shown that such examples can be exploited to attack machine learning systems in safety-critical applications such as autonomous robotics [36] and malware classication [19].…”

Deep neural rejection against adversarial examples

“…Even if the majority of approaches implementing rejection or abstaining classifiers have not considered the problem of defending against adversarial examples, some recent work has explored this direction too [12], [13]. Nevertheless, with respect to the approach proposed in this work, they have only considered the output of the last network layer and perform rejection based solely on that specific feature representation.…”

“…In particular, Bendale and Boult [13] have proposed a rejection mechanism based on reducing the open-set risk in the feature space of the activation vectors extracted from the last layer of the network, while Melis et.al. [12] have applied a threshold on the output of an RBF SVM classifier. Despite these differences, the rationale of the two approaches is quite similar and resembles the older idea of distance-based rejection.…”

Security for Machine Learning-Based Systems: Attacks and Challenges During Training and Inference

“…Unlike the traditional computing algorithms, ML algorithms dynamically change the computational flow with respect to the input data, which increases the energy overhead. Moreover, due to the unpredictability of the computing in hidden layers of neural networks, these algorithms possess several security vulnerabilities which result in increased system vulnerability towards security threats [8]- [11]. Some example are: Amazon echo hacking [8], Facebook chatbots [8], selfdriving bus crashes (on its very first day in Las Vegas) [13].…”