DeepAir: Deep Reinforcement Learning for Adaptive Intrusion Response in Software-Defined Networks

Learning Security Strategies through Game Play and Optimal Stopping

“…Many recent results of automating security strategies have been obtained using reinforcement learning methods. In particular, a large number of studies have focused on intrusion prevention use cases similar to the one we discuss in this paper [13], [14], [26], [27], [19], [25], [28], [20], [29], [21], [22], [24], [83], [84], [85], [23], [86], [33], [35], [34].…”

“…These works use a variety of models, including MDPs [19], [20], [21], [22], [23], [35], Markov games [26], [13], [83], [33], attack graphs [34], and POMDPs [14], [24], [25], as well as various reinforcement learning algorithms, including Q-learning [26], [19], [20], [36], SARSA [25], PPO [13], [14], [34], [35], hierarchical reinforcement learning [21], DQN [22], Thompson sampling [24], MuZero [83], NFQ [84], DDQN [23], NFSP [37], and DDPG [85], [33].…”

“…Some prior work on automated intrusion prevention that make use of emulation are: [83], [84], [85], [87], and [23]. They emulate software-defined networks based on Mininet [88].…”

“…This novel formulation allows us a) to derive structural properties of optimal strategies using results from optimal stopping theory; and b) to find defender strategies that are effective against attackers with dynamic strategies. We thus address a limitation of many related works that consider static attackers only [19], [20], [21], [22], [23], [14], [24], [25]. Second, we propose T-FP, an efficient reinforcement learning algorithm that exploits structural properties of optimal stopping strategies and outperforms a state-of-the-art algorithm for our use case.…”

“…,l ,l (b(1)) = π 1,l (S|b(1)) R st /l − R cost /l (37)− R int + V k−1 π 1,l ,l 1, b(1) + R int + o f O (o|0) V k π 1,l ,l 0, b o (1) − (1 − φ l )f O (o|1)V k π 1,l ,l 1, b o (1) The first term is non-decreasing with π 1,l (S|b(1)) since R st /l − R cost /l − R int ≥ 0 (see Section III) and V k−1 π 1,l ,l 1, b(1)≥ 0 (it is a consequence of Lemma 1 and Eqs [22][23][24][25]…”