Inverse-Inverse Reinforcement Learning. How to Hide Strategy from an Adversarial Inverse Reinforcement Learner

Pattanayak, Kunal; Krishnamurthy, Vikram; Berry, C. P. L.

doi:10.48550/arxiv.2205.10802

Cited by 3 publications

(4 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This paper builds significantly on our previous work [4] on ECM for identifying cognitive radars, and [53], [54], [55] on ECCM for masking radar cognition. Theorem 6 extends IRL for cognitive radars [4] when the radar faces multiple resource constraints.…”

Section: Conclusion and Extensionsmentioning

confidence: 87%

See 1 more Smart Citation

How Can a Cognitive Radar Mask its Cognition?

Pattanayak

Krishnamurthy

Berry

2022

ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

A cognitive radar is a constrained utility maximizer that adapts its sensing mode in response to a changing environment. If an adversary can estimate the utility function of a cognitive radar, it can determine the radar's sensing strategy and mitigate the radar performance via electronic countermeasures (ECM). This paper discusses how a cognitive radar can hide its strategy from an adversary that detects cognition. The radar does so by transmitting purposefully designed sub-optimal responses to spoof the adversary's Neyman-Pearson detector. We provide theoretical guarantees by ensuring the Type-I error probability of the adversary's detector exceeds a pre-defined level for a specified tolerance on the radar's performance loss. We illustrate our cognition masking scheme via numerical examples involving waveform adaptation and beam allocation. We show that small purposeful deviations from the optimal strategy of the radar confuse the adversary by significant amounts, thereby masking the radar's cognition. Our approach uses novel ideas from revealed preference in microeconomics and adversarial inverse reinforcement learning. Our proposed algorithms provide a principled approach for system-level electronic countercountermeasures (ECCM) to mask the radar's cognition, i.e. , hide the radar's strategy from an adversary. We also provide performance bounds for our cognition masking scheme when the adversary has misspecified measurements of the radar's response.

show abstract

Section: Conclusion and Extensionsmentioning

confidence: 87%

“…Theorem 7 generalizes the cognition masking result of [53] to multiple constraints. Our previous works [53], [54], [55] assume optimal adversarial IRL via Afriat's theorem. This paper generalizes cognition masking to sub-optimal adversarial IRL algorithms.…”

Section: Conclusion and Extensionsmentioning

confidence: 99%

How Can a Cognitive Radar Mask its Cognition?

Pattanayak

Krishnamurthy

Berry

2022

ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

show abstract

“…Similarly, in the reinforcement learning community, "inverse reinforcement learning" methods (Arora & Doshi, 2021;Ng, Russell, et al, 2000;Ramachandran & Amir, 2007) can infer reward functions, which can then be applied to influence observers, e.g. to make a robot's motion "legible" to humans (Dragan, 2015;Dragan, Lee, & Srinivasa, 2013;Hadfield-Menell, Russell, Abbeel, & Dragan, 2016) or oppositely to strategically fool adversarial viewers about its true intentions (Pattanayak, Krishnamurthy, & Berry, 2022).…”

Section: Background: Bayesian Inverse Planningmentioning

confidence: 99%

Storytelling as Inverse Inverse Planning

Chandra¹,

Li²,

Tenenbaum³

et al. 2023

Preprint

View full text Add to dashboard Cite

Great storytelling takes us on a journey the way ordinary reality rarely does. But what exactly do we mean by a "journey"? Recently, literary theorist Kukkonen (2014) proposed that storytelling is "probability design": the art of giving an audience pieces of information bit by bit, to craft the journey of their changing beliefs about the fictional world. A good "probability design" choreographs a delicate dance of certainty and surprise in the reader's mind as the story unfolds from beginning to end. In this paper, we computationally model this conception of storytelling. Building on the classic Bayesian inverse planning model of human social cognition, we treat storytelling as inverse inverse planning: the task of choosing actions to manipulate an inverse planner's inferences, and therefore a human audience's beliefs. First, we use an inverse inverse planner to depict social and physical situations, and present behavioral studies indicating that inverse inverse planning produces more expressive behavior than ordinary "naive planning." Then, through a series of examples, we demonstrate how inverse inverse planning captures many storytelling elements from first principles: character, narrative arcs, plot twists, irony, flashbacks, and deus ex machina are all naturally encoded in the flexible language of probability design. This paper reports on work to be presented at SIGGRAPH 2023 (Chandra, Li, Tenenbaum, & Ragan-Kelley, 2023).

show abstract

“…The Rational Communicative Social Actions (RCSA) framework integrates inverse planning and RSA to model communicative aspects of intimacy (Hung, Thomas, Radkani, Tenenbaum, & Saxe, 2022) and punishment . Similarly, in the reinforcement learning community, "inverse reinforcement learning" methods (Arora & Doshi, 2021;Ng, Russell, & others, 2000;Ramachandran & Amir, 2007) can infer reward functions, which can then be applied to influence observers, for example, to make a robot's motion "legible" to humans (Dragan, 2015;Dragan, Lee, & Srinivasa, 2013;Hadfield-Menell, Russell, Abbeel, & Dragan, 2016) or oppositely to strategically fool adversarial viewers about its true intentions (Pattanayak, Krishnamurthy, & Berry, 2022).…”

Section: Background: Bayesian Inverse Planningmentioning

confidence: 99%

Storytelling as Inverse Inverse Planning

Chandra,

Li,

Tenenbaum

et al. 2023

Topics in Cognitive Science

View full text Add to dashboard Cite

Great storytelling takes us on a journey the way ordinary reality rarely does. But what exactly do we mean by this “journey?” Recently, literary theorist Karin Kukkonen proposed that storytelling is “probability design:” the art of giving an audience pieces of information bit by bit, to craft the journey of their changing beliefs about the fictional world. A good “probability design” choreographs a delicate dance of certainty and surprise in the reader's mind as the story unfolds from beginning to end. In this paper, we computationally model this conception of storytelling. Building on the classic Bayesian inverse planning model of human social cognition, we treat storytelling as inverse inverse planning: the task of choosing actions to manipulate an inverse planner's inferences, and therefore a human audience's beliefs. First, we use an inverse inverse planner to depict social and physical situations, and present behavioral studies indicating that inverse inverse planning produces more expressive behavior than ordinary “naïve planning.” Then, through a series of examples, we demonstrate how inverse inverse planning captures many storytelling elements from first principles: character, narrative arcs, plot twists, irony, flashbacks, and deus ex machina are all naturally encoded in the flexible language of probability design.

show abstract

Inverse-Inverse Reinforcement Learning. How to Hide Strategy from an Adversarial Inverse Reinforcement Learner

Cited by 3 publications

References 22 publications

How Can a Cognitive Radar Mask its Cognition?

How Can a Cognitive Radar Mask its Cognition?

Storytelling as Inverse Inverse Planning

Storytelling as Inverse Inverse Planning

Contact Info

Product

Resources

About