Multi-agent Reinforcement Learning in Bayesian Stackelberg Markov Games for Adaptive Moving Target Defense

Sengupta, Sailik; Kambhampati, Subbarao

doi:10.48550/arxiv.2007.10457

Cited by 13 publications

(20 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Markov games or stochastic games are examples of the initial applications of sequential multi-agent games that can be solved using dynamic programming (DP), Q-learning, or linear programming techniques [45]. Uncertainties in an agent's payoff and reward/utility can be also modeled using Bayesian-Stackelberg games [241]. Evolutionary games are other variations of game theory techniques applied for modeling the collective behavior of the agents, with bounded rationality repeatedly looking for equilibrium points [242].…”

Section: Control Techniquesmentioning

confidence: 99%

A Survey of Adaptive Multi-Agent Networks and Their Applications in Smart Cities

Nezamoddini

Gholami

2022

Smart Cities

View full text Add to dashboard Cite

The world is moving toward a new connected world in which millions of intelligent processing devices communicate with each other to provide services in transportation, telecommunication, and power grids in the future’s smart cities. Distributed computing is considered one of the efficient platforms for processing and management of massive amounts of data collected by smart devices. This can be implemented by utilizing multi-agent systems (MASs) with multiple autonomous computational entities by memory and computation capabilities and the possibility of message-passing between them. These systems provide a dynamic and self-adaptive platform for managing distributed large-scale systems, such as the Internet-of-Things (IoTs). Despite, the potential applicability of MASs in smart cities, very few practical systems have been deployed using agent-oriented systems. This research surveys the existing techniques presented in the literature that can be utilized for implementing adaptive multi-agent networks in smart cities. The related literature is categorized based on the steps of designing and controlling these adaptive systems. These steps cover the techniques required to define, monitor, plan, and evaluate the performance of an autonomous MAS. At the end, the challenges and barriers for the utilization of these systems in current smart cities, and insights and directions for future research in this domain, are presented.

show abstract

Section: Control Techniquesmentioning

confidence: 99%

A Survey of Adaptive Multi-Agent Networks and Their Applications in Smart Cities

Nezamoddini

Gholami

2022

Smart Cities

View full text Add to dashboard Cite

show abstract

“…Moving Target Defense (MTD) is one of the modern technologies to neutralize attacker's position advantage by creating reconnaissance difficulties and uncertainties for attackers. There is a surge of recent literature on using RL to choose an adaptive configuration strategy to maximize the impact of MTD with particular focuses on the dynamic environment [63,64], reduced resource consumption [65], usability [11], partially observable environment [66,67], and multiagent scenarios that contains both the characteristics of the system and the adversary's observed activities [68,67].…”

Section: Posture-related Vulnerabilitymentioning

confidence: 99%

Reinforcement Learning for Feedback-Enabled Cyber Resilience

Huang¹,

Huang²,

Zhu³

2021

Preprint

View full text Add to dashboard Cite

The rapid growth in the number of devices and their connectivity has enlarged the attack surface and made cyber systems more vulnerable. As attackers become increasingly sophisticated and resourceful, mere reliance on traditional cyber protection, such as intrusion detection, firewalls, and encryption, is insufficient to secure the cyber systems. Cyber resilience provides a new security paradigm that complements inadequate protection with resilience mechanisms. A Cyber-Resilient Mechanism (CRM) adapts to the known or zero-day threats and uncertainties in real-time and strategically responds to them to maintain the critical functions of the cyber systems in the event of successful attacks. Feedback architectures play a pivotal role in enabling the online sensing, reasoning, and actuation process of the CRM. Reinforcement Learning (RL) is an important class of algorithms that epitomize the feedback architectures for cyber resiliency. It allows the CRM to provide dynamic and sequential responses to attacks with limited or without prior knowledge of the environment and the attacker. In this work, we review the literature on RL for cyber resiliency and discuss the cyber-resilient defenses against three major types of vulnerabilities, i.e., posture-related, information-related, and human-related vulnerabilities. We introduce moving target defense, defensive cyber deception, and assistive human security technologies as three application domains of CRMs to elaborate on their designs. The RL algorithms also have vulnerabilities themselves. We explain the major vulnerabilities of RL and present works that develop several attack models, in which the attacks target the rewards, the state observations, and the action commands. We show that the attacker can trick the RL agent into learning a nefarious policy with minimum attacking effort. We discuss the potential defense methods to secure them and find that there is a lack of works focusing on the defensive mechanisms for RL-enabled systems. Finally, we discuss the future challenges of RL for cyber security and resiliency and emerging applications of RL-based CRMs.

show abstract

“…Although most papers have employed frameworks where agents are taking actions simultaneously, authors in [7,2] used the Stackelberg games framework as a solution for certain wireless sensor network resource management problem.…”

Section: Stackelberg Gamesmentioning

confidence: 99%

Survey on Multi-Agent Q-Learning frameworks for resource management in wireless sensor network

Tashakori

2021

Preprint

View full text Add to dashboard Cite

This report aims to survey multi-agent Q-Learning algorithms, analyze different game theory frameworks used, address each framework's applications, and report challenges and future directions. The target application for this study is resource management in the wireless sensor network.In the first section, the author provided an introduction regarding the applications of wireless sensor networks. After that, the author presented a summary of the Q-Learning algorithm, a well-known classic solution for model-free reinforcement learning problems.In the third section, the author extended the Q-Learning algorithm for multi-agent scenarios and discussed its challenges.In the fourth section, the author surveyed sets of game-theoretic frameworks that researchers used to address this problem for resource allocation and task scheduling in the wireless sensor networks. Lastly, the author mentioned some interesting open challenges in this domain.Wireless sensor network provides online monitoring capabilities in situations that are not accessible (for example: controlling the temperature of a nuclear reactor, or invasive brain, or muscular signal monitoring).Usually, Wireless sensor nodes are heterogeneous, energy-constrained, and tend to operate in dynamic and unclear situations. In these situations, nodes need to learn how to cooperate over tasks and resources (including power and bandwidth). It implies that we design a framework that allows wireless sensor nodes to adapt to the new situation. In these scenarios, Reinforcement Learning is an immeasurable solution. [8].Recently, reinforcement learning (RL) becomes a trend in various autonomous decision-making tasks, whether sequential or simultaneous. For example, tasks related to solving strategic games, or sensor and communication networks, finances, social science, etc. [12].

show abstract

Multi-agent Reinforcement Learning in Bayesian Stackelberg Markov Games for Adaptive Moving Target Defense

Cited by 13 publications

References 22 publications

A Survey of Adaptive Multi-Agent Networks and Their Applications in Smart Cities

A Survey of Adaptive Multi-Agent Networks and Their Applications in Smart Cities

Reinforcement Learning for Feedback-Enabled Cyber Resilience

Survey on Multi-Agent Q-Learning frameworks for resource management in wireless sensor network

Contact Info

Product

Resources

About