Adaptive Cyber Defense Against Multi-Stage Attacks Using Learning-Based POMDP

Abstract

Growing multi-stage attacks in computer networks impose significant security risks and necessitate the development of effective defense schemes that are able to autonomously respond to intrusions during vulnerability windows. However, the defender faces several real-world challenges, e.g., unknown likelihoods and unknown impacts of successful exploits. In this article, we leverage reinforcement learning to develop an innovative adaptive cyber defense to maximize the cost-effectiveness subject to the aforementioned challenges. In particular, we use Bayesian attack graphs to model the interactions between the attacker and networks. Then we formulate the defense problem of interest as a partially observable Markov decision process problem where the defender maintains belief states to estimate system states, leverages Thompson sampling to estimate transition probabilities, and utilizes reinforcement learning to choose optimal defense actions using measured utility values. The algorithm performance is verified via numerical simulations based on real-world attacks.

Learning Security Strategies through Game Play and Optimal Stopping

“…Many recent results of automating security strategies have been obtained using reinforcement learning methods. In particular, a large number of studies have focused on intrusion prevention use cases similar to the one we discuss in this paper [13], [14], [26], [27], [19], [25], [28], [20], [29], [21], [22], [24], [83], [84], [85], [23], [86], [33], [35], [34].…”

“…These works use a variety of models, including MDPs [19], [20], [21], [22], [23], [35], Markov games [26], [13], [83], [33], attack graphs [34], and POMDPs [14], [24], [25], as well as various reinforcement learning algorithms, including Q-learning [26], [19], [20], [36], SARSA [25], PPO [13], [14], [34], [35], hierarchical reinforcement learning [21], DQN [22], Thompson sampling [24], MuZero [83], NFQ [84], DDQN [23], NFSP [37], and DDPG [85], [33].…”

“…Third, our method to find effective defender strategies includes using an emulation system in addition to a simulation system. The advantage of our method compared to the simulation-only approaches [13], [14], [26], [27], [19], [25], [28], [20], [29], [21], [22], [24], [33], [34], [35], [36], [37] is that the parameters of our simulation system are determined by measurements from an emulation system instead of being chosen by a human expert. Further, the learned strategies are evaluated in the emulation system, not in the simulation system.…”

“…This novel formulation allows us a) to derive structural properties of optimal strategies using results from optimal stopping theory; and b) to find defender strategies that are effective against attackers with dynamic strategies. We thus address a limitation of many related works that consider static attackers only [19], [20], [21], [22], [23], [14], [24], [25]. Second, we propose T-FP, an efficient reinforcement learning algorithm that exploits structural properties of optimal stopping strategies and outperforms a state-of-the-art algorithm for our use case.…”

“…Third, we provide evaluation results from an emulated infrastructure. This addresses a common drawback in related research, which relies on simulations to learn and evaluate strategies [13], [14], [26], [27], [19], [25], [28], [20], [29], [21], [22], [24], [11], [30], [31], [32], [33], [34], [35], [36], [37].…”