Reinforcement Learning vs Genetic Algorithms in Game-Theoretic Cyber-Security

Abstract

Penetration testing is the practice of performing a simulated attack on a computer system in order to reveal its vulnerabilities. The most common approach is to gain information and then plan and execute the attack manually, by a security expert. This manual method cannot meet the speed and frequency required for efficient, large-scale secu- rity solutions development. To address this, we formalize penetration testing as a security game between an attacker who tries to compro- mise a network and a defending adversary actively protecting it. We compare multiple algorithms for finding the attacker’s strategy, from fixed-strategy to Reinforcement Learning, namely Q-Learning (QL), Extended Classifier Systems (XCS) and Deep Q-Networks (DQN). The attacker’s strength is measured in terms of speed and stealthi- ness, in the specific environment used in our simulations. The results show that QL surpasses human performance, XCS yields worse than human performance but is more stable, and the slow convergence of DQN keeps it from achieving exceptional performance, in addition, we find that all of these Machine Learning approaches outperform fixed-strategy attackers.

Research and Challenges of Reinforcement Learning in Cyber Defense Decision-Making for Intranet Security

“…Niculae built a more realistic multi-agent system, where included the interactions among attackers, defenders and normal users [56]. In the system, ten attack actions, three defense actions and three normal user actions were designed.…”

Exploring SQL Injection Vulnerabilities Using Artificial Bee Colony

“…Manual penetration tests, although effective, can hardly meet all security requirements that are constantly changing and evolving (Niculae, 2018). Furthermore, they require specialized knowledge which, in addition to presenting a high cost, is typically slower.…”

“…The alternative is automated tools that, although faster, often do not adapt to the context and uniqueness of each application. In (Niculae, 2018), the author developed a reinforcement learning strategy capable of compromising a system faster than a brute force and random approach. This concluded that it is possible to build an agent capable of learning and evolving over time so that it can penetrate a network.…”

CybORG: An Autonomous Cyber Operations Research Gym

“…Pen testing simulations [17,20] have been used to train red agents using reinforcement learning. These simulations use a realistic action space to enable agents to learn how to attack a network.…”