Guarded Deep Learning using Scenario-based Modeling

Abstract

Deep neural networks (DNNs) are becoming prevalent, often outperforming manually-created systems. Unfortunately, DNN models are opaque to humans, and may behave in unexpected ways when deployed. One approach for allowing safer deployment of DNN models calls for augmenting them with hand-crafted override rules, which serve to override decisions made by the DNN model when certain criteria are met. Here, we propose to bring together DNNs and the well-studied scenario-based modeling paradigm, by expressing these override rules as simple and intuitive scenarios. This approach can lead to override rules that are comprehensible to humans, but are also sufficiently expressive and powerful to increase the overall safety of the model. We describe how to extend and apply scenario-based modeling to this new setting, and demonstrate our proposed technique on multiple DNN models.

Towards Repairing Scenario-Based Models with Rich Events

“…Although vanilla SBM has been successfully used in various contexts, in recent years it was shown that it may fall short in expressing various complex interactions between scenario objects (Katz et al, 2019;Katz, 2020;Elyasaf, 2020). Specifically, the simple event declaration mechanism -a finite set of events E, and a finite set of requested, waited-for and blocked events in every state, may be inadequate for expressing more complex behaviors.…”

“…Recently, it has been observed that while SBM is well equipped for modeling reactive systems, it is sometimes inadequate for modeling systems that handle data -for example, robotics and autonomous vehicle systems (Katz et al, 2019;Katz, 2020), which involve various mathematical computations in addition to their reactivity. To this end, researchers have extended the SBM principles, allowing the event selection mechanism to support rich events, i.e.…”

Towards combining deep learning, verification, and scenario-based programming

“…In another recent project [13], we demonstrated how SBP can be used for guarding DNN-based systems. Specifically, we targeted state-of-the-art computer network systems, and demonstrated how manually-crafted scenarios could be used to correct various bugs discovered, e.g., through verification [16].…”

“…choose a low sending rate even though higher rates could be used. Thus, we augmented this system with SBP components that ensured that this did not happen, by identifying such a situation and forcing the system to try out a higher sending rate [13]. In another example, we studied a DNN-based resource manager [17]: a system that controls CPU and memory resources, and assigns them to incoming jobs in order to maximize throughout.…”

“…DNN verification has revealed that this system suffers from certain security vulnerabilities [16], and that specific inputs could cause it to repeatedly assign resources to non-existing jobs, effectively starving other, legitimate jobs. In [13,14] we showed how the system could be augmented with SBP components that generalized, and prevented, such undesirable behavior.…”