Framework for Complex SoS Emergent Behavior Evolution Using Deep Reinforcement Learning

Abstract

Advances in technology have made it easy to integrate multiple modern systems to form complex system‐of‐systems (SoS) to achieve unparalleled levels of functionality that are otherwise not achievable by the constituent systems in isolation. In fact, with the recent explosion of machine learning techniques to build autonomous systems such as drones and self‐driving cars, there is a pressing need to ensure that they collaboratively and safely operate in an SoS context. However, in general, the characteristic emergent behaviors of complex SoS – that directly impact its operational measures of success or Measures of Effectiveness (MOEs) – is very difficult, if not impossible, to manually explore, anticipate, and arbitrate just from knowledge of its underlying systems. Further, there are multiple scenarios of evolution in such complex SoS, including evolution in the emergent behavior of the SoS. The continuous, continual, and evolving nature of the SoS and constituent system environment's state and possible actions, adds further complexity. In this paper, we present a novel approach that leverages Reinforcement Learning, a machine learning approach, to inculcate adaptable intelligence in constituent systems to adapt their behaviors in tandem with the evolution of emergent behavior at the SoS level. By augmenting the reward mechanism of RL by leveraging SoS‐Constituent System MOE Relationship, that relates and ranks System MOEs vs. SoS MOEs, we inculcate an Intelligent‐Behavior Evolution Agent, with the necessary constraints to learn to maximize the SoS and system‐level MOEs, while adapting itself to the evolution in SoS. We illustrate our approach and demonstrate its feasibility and potential by applying it to a power grid SoS case example. The effectiveness and performance of the approach are quantified.

Detecting emergence in engineered systems: A literature review and synthesis approach

“…56 Raz et al 56 Raman and Jeppu. [57][58][59] Raman et al 60 and Raman and Murugesan 61,62 state that modeling can help us regarding simple emergence, while simulation is required to detect weak emergence. We can update our modeling efforts when we understand the simulated emergent behavior, to evolve the system design and exploit and/or mitigate the detected emergence.…”

“…Many methods have shown promising results in different case studies. [1][2][3]17,28,30,36,[49][50][51][52][53][54][56][57][58][59][60][61][62][63][64][65][66][67]70,71,[73][74][75][76][77][78][79][80] The approaches range from intuitive specialized model views used by Guariniello et al 1 to advanced non-intuitive methods like Machine Learning used by Raz et al 56 Different approaches will serve different cases to a varying degree, requiring a thorough evaluation process to select the most appropriate method to utilize in each case. In the following, we will discuss the proposed research questions.…”

“…49 The second approach is related to how efficient we are able to execute a test cycle, looking into the manual versus automatic task balance. 17,28,36,54,80 Machine Learning is a method of interest to automate data analysis, [56][57][58][59][60][61][62] and statistical debugging is another. 79 The third approach is modeling and simulation to increase our level of understanding through simulation and updated models.…”

“…Raman and Jeppu, [57][58][59] Raman et al, 60 and Raman and Murugesan 61,62 The Measures of Performance (MoP's) and Measures of Effectiveness (MoE's) can be monitored using Machine Learning to look for changes that give or could give raise to emergent behaviors. Machine Learning could be used to adapt system behavior in tandem with the evolution of emergent behavior in a complex system-of-systems.…”

“…For a complex system, we might need to move away from the traditional test coverage approach and look more into incentives to adapt the system behavior. 4,5,61 No amount of testing can guarantee that a complex system will not fail, as well as the number of permutations rendering the calculations and storage impossible. 55 Complicated systems can be seen as sub-systems of a complex system-of-systems, and are the bedrock of Systems Engineering practice.…”