Key Concepts in AI Safety: Robustness and Adversarial Examples

Abstract

This paper is the second installment in a series on “AI safety,” an area of machine learning research that aims to identify causes of unintended behavior in machine learning systems and develop tools to ensure these systems work safely and reliably. The first paper in the series, “Key Concepts in AI Safety: An Overview,” described three categories of AI safety issues: problems of robustness, assurance, and specification. This paper introduces adversarial examples, a major challenge to robustness in modern machine learning systems.

Learning stability of partially observed switched linear systems

“…Theorem 3 (Main result) Consider the SLS Σ as given in (1), where the initial state is i.i.d with the uniform distribution over the unit sphere S n−1 and the switching signal is uniform and i.i.d. in M. Given N, T ∈ Z + , the sample set ω N as defined in (4), and k ≤ T − 1, let (γ * k (ω N ), P * k (ω N )) be the unique solution of the scenario program (9).…”

“…holds for any output sequence (y(0), y(1), • • • , y(T − 1)) and generated by (1). Hence, in order to find a bound of the JSR, we have to solve the optimization problem for some sufficiently large…”

“…Theorem 2 Consider a pathwise observable SLS Σ as in (1).Suppose there exist k ∈ Z + , ∈ Z + , γ ≥ 0, P 0 and a subset S ⊆ S n−1 such that, k ≥ H(Σ), and for any σ σ σ ∈ M k+ , x A σ σ σ 0: −1 P σ σ σ Σ (P, , k)A σ σ σ 0: −1 x ≤γ 2 x P σ σ σ Σ (P, 0, k)x, ∀x ∈ S n−1 \ S.…”

“…Verification of safety and robustness of AI systems has gained significant attention in recent years [1,2,3]. This is particularly important in the context of application of machine learning algorithms to cyber-physical systems (CPS), which tend to be safety-critical (e.g., autonomous vehicles, etc.…”

Machine Learning and Cybersecurity: Hype and Reality

“…The process by which many ML systems reach decisions can often be poorly understood and highly sensitive to small changes that a human analyst would view as trivial, which often makes it possible for attackers to find "adversarial examples"-slightly altered inputs that dramatically change a model's response despite being undetectable to a human. 52 The use of ML models also opens up new avenues of attack: the model itself must be kept secure, but defenders must also make sure that their data is not poisoned and that the (typically open source) algorithms and statistical packages they use have not been tampered with. 53 In addition, while machine learning is sometimes presented as an objective process of "learning patterns from data," in reality the design of ML systems is often the result of many judgment calls.…”