Compact Representation of Value Function in Partially Observable Stochastic Games

Horák, Karel; Bošanský, Branislav; Kiekintveld, Christopher; Kamhoua, Charles A.

doi:10.24963/ijcai.2019/50

Cited by 12 publications

(4 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Direct methods such as dynamic programming are intractable in practice [13]. We list the following approaches that can compute policies on small POSGs: using a memory bounded representation of the value function [11]; heuristically reformulating as a Bayesian game [26]; performing value iteration and applying beliefbased heuristics [16]; and leveraging marginal probabilities to compactly represent the belief spaces [18] . While these general approaches rely on fewer assumptions than the proposed, dedicated, method, they are only tractable for model sizes of less than 10 6 states.…”

Section: B Related Workmentioning

confidence: 99%

Safe Policies for Factored Partially Observable Stochastic Games

Carr¹,

Jansen²,

Bharadwaj³

et al. 2021

Robotics: Science and Systems XVII

View full text Add to dashboard Cite

We study planning problems where a controllable agent operates under partial observability and interacts with an uncontrollable opponent, also referred to as the adversary. The agent has two distinct objectives: To maximize an expected value and to adhere to a safety specification. Multi-objective partially observable stochastic games (POSGs) formally model such problems. Yet, even for a single objective, the task of computing suitable policies for POSGs is theoretically hard and computationally intractable in practice. Using a factored state-space representation, we define a decoupling scheme for the POSG state space that-under certain assumptions on the observability and the reward structure-separates the state components relevant for the reward from those relevant for safety. This decoupling affects the possibility to compute provably safe and reward-optimal policies in a tractable two-stage approach. In particular, on the fully observable components related to safety, we exactly compute the set of policies that captures all possible safe choices against the opponent. We restrict the agent's behavior to these safe policies and project the POSG to a partially observable Markov decision process (POMDP). Any reward-maximal policy for the POMDP is then guaranteed to be safe and reward-maximal for the POSG. We showcase our approach's feasibility using high-fidelity simulations of two case studies that concern UAV path planning and autonomous driving. Moreover, to demonstrate the practical applicability, we design a physical experiment involving a robot decision making problem under energy constraints that is motivated by a paired helicopter with NASA's Perseverance Mars rover.

show abstract

Section: B Related Workmentioning

confidence: 99%

Safe Policies for Factored Partially Observable Stochastic Games

Carr¹,

Jansen²,

Bharadwaj³

et al. 2021

Robotics: Science and Systems XVII

View full text Add to dashboard Cite

show abstract

“…loss) functions, the player's strategies are computed by maximizing (resp. minimizing) the objective function [8], [11], [13]. Carrol and Grosu [5] used a signaling game to study honeypot deception.…”

Section: Introductionmentioning

confidence: 99%

Secure-by-synthesis network with active deception and temporal logic specifications

Fu,

Kulkarni,

Luo

et al. 2020

Preprint

View full text Add to dashboard Cite

This paper is concerned with the synthesis of strategies in network systems with active cyber deception. Active deception in a network employs decoy systems and other defenses to conduct defensive planning against the intrusion of malicious attackers who have been confirmed by sensing systems. In this setting, the defender's objective is to ensure the satisfaction of security properties specified in temporal logic formulas. We formulate the problem of deceptive planning with decoy systems and other defenses as a two-player games with asymmetrical information and Boolean payoffs in temporal logic. We use level-2 hypergame with temporal logic objectives to capture the incomplete/incorrect knowledge of the attacker about the network system as a payoff misperception. The true payoff function is private information of the defender. Then, we extend the solution concepts of ω-regular games to analyze the attacker's rational strategy given her incomplete information. By generalizing the solution of level-2 hypergame in the normal form to extensive form, we extend the solutions of games with safe temporal logic objectives to decide whether the defender can ensure security properties to be satisfied with probability one, given any possible strategy that is perceived to be rational by the attacker. Further, we use the solution of games with co-safe (reachability) temporal logic objectives to determine whether the defender can engage the attacker, by directing the attacker to a high-fidelity honeypot. The effectiveness of the proposed synthesis methods is illustrated with synthetic network systems with honeypots.

show abstract

“…Active deception [Jajodia et al, 2016] employs decoy systems and other defenses, including access control and online network reconfiguration, to conduct deceptive planning against the intrusion of malicious users who have been detected and confirmed by sensing systems. To design defense strategies with deception, game theory has been employed [Hor, 2012, Al-Shaer et al, 2019, Horak et al, 2019, Cohen, 2006, Zhu and Rass, 2018. These game-theoretic models express the attacker and defender's objectives using reward/loss functions.…”

Section: Introductionmentioning

confidence: 99%

“…These game-theoretic models express the attacker and defender's objectives using reward/loss functions. In [Horak et al, 2019], a partially observable stochastic game is formulated to capture the interaction between an attacker and a defender with one-sided partial observations. The attacker is to exploit and compromise the system without being detected and has complete observation.…”

Section: Introductionmentioning

confidence: 99%

A Theory of Hypergames on Graphs for Synthesizing Dynamic Cyber Defense with Deception

Kulkarni

2020

Preprint

View full text Add to dashboard Cite

In this chapter, we present an approach using formal methods to synthesize reactive defense strategy in a cyber network, equipped with a set of decoy systems. We first generalize formal graphical security models-attack graphs-to incorporate defender's countermeasures in a game-theoretic model, called an attack-defend game on graph. This game captures the dynamic interactions between the defender and the attacker and their defense/attack objectives in formal logic. Then, we introduce a class of hypergames to model asymmetric information created by decoys in the attacker-defender interactions. Given qualitative security specifications in formal logic, we show that the solution concepts from hypergames

show abstract

Compact Representation of Value Function in Partially Observable Stochastic Games

Cited by 12 publications

References 2 publications

Safe Policies for Factored Partially Observable Stochastic Games

Safe Policies for Factored Partially Observable Stochastic Games

Secure-by-synthesis network with active deception and temporal logic specifications

A Theory of Hypergames on Graphs for Synthesizing Dynamic Cyber Defense with Deception

Contact Info

Product

Resources

About