Inverse-Inverse Reinforcement Learning. How to Hide Strategy from an Adversarial Inverse Reinforcement Learner

Pattanayak, Kunal; Krishnamurthy, Vikram; Berry, Christopher

doi:10.1109/cdc51059.2022.9992959

Cited by 7 publications

(1 citation statement)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Rational Communicative Social Actions (RCSA) framework integrates inverse planning and RSA to model communicative aspects of intimacy (Hung, Thomas, Radkani, Tenenbaum, & Saxe, 2022) and punishment (Radkani, Tenenbaum, & Saxe, 2022). Similarly, in the reinforcement learning community, “inverse reinforcement learning” methods (Arora & Doshi, 2021; Ng, Russell, & others, 2000; Ramachandran & Amir, 2007) can infer reward functions, which can then be applied to influence observers, for example, to make a robot's motion “legible” to humans (Dragan, 2015; Dragan, Lee, & Srinivasa, 2013; Hadfield‐Menell, Russell, Abbeel, & Dragan, 2016) or oppositely to strategically fool adversarial viewers about its true intentions (Pattanayak, Krishnamurthy, & Berry, 2022).…”

Section: Background: Bayesian Inverse Planningmentioning

confidence: 99%

Storytelling as Inverse Inverse Planning

Chandra,

Li,

Tenenbaum

et al. 2023

Topics in Cognitive Science

View full text Add to dashboard Cite

Great storytelling takes us on a journey the way ordinary reality rarely does. But what exactly do we mean by this “journey?” Recently, literary theorist Karin Kukkonen proposed that storytelling is “probability design:” the art of giving an audience pieces of information bit by bit, to craft the journey of their changing beliefs about the fictional world. A good “probability design” choreographs a delicate dance of certainty and surprise in the reader's mind as the story unfolds from beginning to end. In this paper, we computationally model this conception of storytelling. Building on the classic Bayesian inverse planning model of human social cognition, we treat storytelling as inverse inverse planning: the task of choosing actions to manipulate an inverse planner's inferences, and therefore a human audience's beliefs. First, we use an inverse inverse planner to depict social and physical situations, and present behavioral studies indicating that inverse inverse planning produces more expressive behavior than ordinary “naïve planning.” Then, through a series of examples, we demonstrate how inverse inverse planning captures many storytelling elements from first principles: character, narrative arcs, plot twists, irony, flashbacks, and deus ex machina are all naturally encoded in the flexible language of probability design.

show abstract