Differentially Private Reinforcement Learning with Linear Function Approximation

Abstract

Motivated by the wide adoption of reinforcement learning (RL) in real-world personalized services, where users' sensitive and private information needs to be protected, we study regret minimization in finite horizon Markov decision processes (MDPs) under the constraints of differential privacy (DP). Compared to existing private RL algorithms that work only on tabular finite-state, finite-actions MDPs, we take the first step towards privacy-preserving learning in MDPs with large state and action spaces. Specifically, we consider MDPs with linear function approximation (in particular linear mixture MDPs) under the notion of joint differential privacy (JDP), where the RL agent is responsible for protecting users' sensitive data. We design two private RL algorithms that are based on value iteration and policy optimization, respectively, and show that they enjoy sub-linear regret performance while guaranteeing privacy protection. Moreover, the regret bounds are independent of the number of states, and scale at most logarithmically with the number of actions, making the algorithms suitable for privacy protection in nowadays large scale personalized services. Our results are achieved via a general procedure for learning in linear mixture MDPs under changing regularizers, which not only generalizes previous results for non-private learning, but also serves as a building block for general private reinforcement learning.

New challenges in reinforcement learning: a survey of security and privacy

“…Intelligent algorithms can handle more complex problems and situations, even if little information about the model and the goal is available; thus, they can expand the horizons of privacy protection. We can further observe that inverse reinforcement learning has been adopted to address these privacy problems [71,72]. In fact, however, inverse reinforcement learning is more frequently used to approach the problem of inferring an expert's reward function from demonstrations and to provide the reward to the learning system rather than to tackle security and/or privacy issues.…”

“…Zhou et al [71] proposed to protect users' sensitive and private information by considering regret minimization in large state and action spaces. Their work used the notion of joint differential privacy (JDP) and considered MDPs by means of linear function approximation.…”