An Approach for Formal Verification of Machine Learning based Complex Systems

Abstract

A complex system is characterized by emergence of global properties which are very difficult, if not impossible, to anticipate just from complete knowledge of component behaviors. Emergence, hierarchical organization and numerosity are some of the characteristics of complex systems. With increasing system complexity, achieving confidence in systems becomes increasingly difficult. With the recent trend towards significant footprint of complex system's functionality being governed by machine learning based models and algorithms, there is a need to ensure that emergent behavior associated with such systems are well analyzed and understood. Traditional verification and validation approaches are often inadequate to bring in the nuances of potential emergent behavior, which may be positive or negative. This paper describes a novel approach towards application of formal methods for analyzing and evaluating emergent behavior of complex systems that are governed by machine learning models. The proposed approach involves developing a machine learning classifier model that learns on potential negative and positive emergent behaviors, and leveraging the classifier in a formal verification model checking environment to assert negative emergent behavior. The approach is illustrated through an example of a pitch control system of an aircraft. The effectiveness and performance of the approach are quantified.

Detecting emergence in engineered systems: A literature review and synthesis approach

“…56 Raz et al 56 Raman and Jeppu. [57][58][59] Raman et al 60 and Raman and Murugesan 61,62 state that modeling can help us regarding simple emergence, while simulation is required to detect weak emergence. We can update our modeling efforts when we understand the simulated emergent behavior, to evolve the system design and exploit and/or mitigate the detected emergence.…”

“…Many methods have shown promising results in different case studies. [1][2][3]17,28,30,36,[49][50][51][52][53][54][56][57][58][59][60][61][62][63][64][65][66][67]70,71,[73][74][75][76][77][78][79][80] The approaches range from intuitive specialized model views used by Guariniello et al 1 to advanced non-intuitive methods like Machine Learning used by Raz et al 56 Different approaches will serve different cases to a varying degree, requiring a thorough evaluation process to select the most appropriate method to utilize in each case. In the following, we will discuss the proposed research questions.…”

“…Raman and Jeppu, [57][58][59] Raman et al, 60 and Raman and Murugesan 61,62 The Measures of Performance (MoP's) and Measures of Effectiveness (MoE's) can be monitored using Machine Learning to look for changes that give or could give raise to emergent behaviors. Machine Learning could be used to adapt system behavior in tandem with the evolution of emergent behavior in a complex system-of-systems.…”

Framework for Formal Verification of Machine Learning Based Complex System‐of‐Systems

“…In this work, we have used Simulink Design Verifier, a tool from MathWorks (SLDV 2020) that uses formal methods to generate test cases, find design errors and to prove the correctness of assertions or properties defined as Simulink blocks or MATLAB code. We have successfully demonstrated the use of SLDV in our earlier work (Raman and Jeppu 2019). In this paper, we also look at another tool called CBMC (2020).…”

Framework for Formal Verification of Machine Learning Based Complex System‐of‐System

“…In this work, we have used Simulink Design Verifier, a tool from MathWorks (SLDV, 2020) that uses formal methods to generate test cases, find design errors and to prove the correctness of assertions or properties defined as Simulink blocks or MATLAB code. We have successfully demonstrated the use of SLDV in our earlier work (Raman and Jeppu, 2019). In this paper, we also look at another tool called CBMC (CBMC, 2020).…”