How Private Is Your RL Policy? An Inverse RL Based Analysis Framework

Abstract

Reinforcement Learning (RL) enables agents to learn how to perform various tasks from scratch. In domains like autonomous driving, recommendation systems and more, optimal RL policies learned could cause a privacy breach if the policies memorize any part of the private reward. We study the set of existing differentially-private RL policies derived from various RL algorithms such as Value Iteration, Deep Q Networks, and Vanilla Proximal Policy Optimization. We propose a new Privacy-Aware Inverse RL (PRIL) analysis framework, that performs reward reconstruction as an adversarial attack on private policies that the agents may deploy. For this, we introduce the reward reconstruction attack, wherein we seek to reconstruct the original reward from a privacy-preserving policy using an Inverse RL algorithm. An adversary must do poorly at reconstructing the original reward function if the agent uses a tightly private policy. Using this framework, we empirically test the effectiveness of the privacy guarantee offered by the private algorithms on multiple instances of the FrozenLake domain of varying complexities. Based on the analysis performed, we infer a gap between the current standard of privacy offered and the standard of privacy needed to protect reward functions in RL. We do so by quantifying the extent to which each private policy protects the reward function by measuring distances between the original and reconstructed rewards.Recent advancements in reinforcement learning (RL) have found widespread application in many real-world domains. Often, these domains are built from rich data sources or real-world environments, which could contain sensitive information of many individuals. This is evident in domains such as autonomous driving, recommendation systems, trading, industrial assembly, and domestic service robots. For example, a recommendation system agent for an online shopping platform not only tracks the purchases made, but also how long the user hovers over an item that he/she did not purchase. Another example is when an autonomous driving agent not only learns the dynamics behind driving, but also identifies people and predicts their behaviours on the streets, and how to respond to such situations. The reward function in these environments is often sensitive, as it is built on people's private information. RL agents trained in such environments should not expose the private information of individuals. We therefore, need to use privacy-preserving methods to protect the rewards from being memorized in the agent's policy in such a manner that the agent's utility is not compromised.Recent works use Differential Privacy (DP) [7] to make the agent's policy quantifiably private and to get a rigorous privacy guarantee. Like many other areas in AI, RL has also started adopting DP to establish a mathematical way of guaranteeing data privacy in RL environments. However, an important question is: does the privacy guarantee offered by the private policies translate well into protecting the reward function? If not, how ca...

New challenges in reinforcement learning: a survey of security and privacy

“…However, the concept of "inferring a reward function" naturally prompts thoughts of privacy leakage and unreliable models. Consequently, it might be possible to consider the security and privacy problems of inverse reinforcement learning, like the work in [76]. Besides the security and problems in reinforcement learning itself, there are several works about applying reinforcement learning to security and privacy problems.…”

“…As differential privacy is always adopted to establish a mathematical way of guaranteeing data privacy in reinforcement learning, and considering that inverse reinforcement learning is applied to inferring the reward function from demonstrations and providing rewards to the learning system, Prakash et al [76] investigated the existing set of privacy techniques for reinforcement learning and proposed a new Privacy-Aware Inverse reinforcement Learning (PRIL) analysis framework, which is a new form of privacy attack that targets the private reward function. This reward attack attempts to reconstruct the original reward from a privacy-preserving policy (such as differential privacy) using an inverse reinforcement algorithm.…”

“…However, it also may be used to attack a system to infer the reward function. Moreover, to obtain rewards, IRL also needs data to train; as a result, privacy leakage problems may arise [76].…”