Locally Differentially Private (Contextual) Bandits Learning

Zheng, Kai; Cai, Tianle; Huang, Weiran; Li, Zhenguo; Wang, Liwei

doi:10.48550/arxiv.2006.00701

Cited by 10 publications

(20 citation statements)

References 13 publications

(22 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Finally, it will also be interesting to consider other privacy guarantees beyond JDP and LDP, e.g., shuffle model of DP (Cheu et al, 2019), that achieves a smooth transition between JDP and LDP. This is particularly useful even in the simpler linear bandit setting as the regret under LDP is O(T 3/4 ) (Zheng et al, 2020) while the regret under JDP is O( √ T ) (Shariff and Sheffet, 2018). Thus, one interesting question is whether the shuffle model of DP can be utilized to achieve a similar regret as in JDP while providing the same strong privacy guarantee as in LDP, which is one of ongoing works.…”

Section: Discussionmentioning

confidence: 99%

“…Recently, another variant of DP, called local differential privacy (LDP) (Duchi et al, 2013) has gained increasing popularity in personalized services due to its stronger privacy protection. It has been studied in various bandit settings recently (Ren et al, 2020;Zheng et al, 2020;Zhou and Tan, 2020;Dubey, 2021). Under LDP, each user's raw data is directly protected before being sent to the learning agent.…”

Section: Related Workmentioning

confidence: 99%

“…LDP ensures that if any adversary (can be the RL agent itself) observes the output of the privacy mechanism M ′ for two different trajectories, then it is statistically difficult for it to guess which output is from which trajectory. This has been used extensively in multi-armed bandits (Zheng et al, 2020;Ren et al, 2020), and introduced in private tabular RL by Garcelon et al (2020).…”

Section: On a Single User And For All Subset Of Actionsmentioning

confidence: 99%

See 2 more Smart Citations

Differentially Private Reinforcement Learning with Linear Function Approximation

Zhou¹

2022

Preprint

View full text Add to dashboard Cite

Motivated by the wide adoption of reinforcement learning (RL) in real-world personalized services, where users' sensitive and private information needs to be protected, we study regret minimization in finite horizon Markov decision processes (MDPs) under the constraints of differential privacy (DP). Compared to existing private RL algorithms that work only on tabular finite-state, finite-actions MDPs, we take the first step towards privacy-preserving learning in MDPs with large state and action spaces. Specifically, we consider MDPs with linear function approximation (in particular linear mixture MDPs) under the notion of joint differential privacy (JDP), where the RL agent is responsible for protecting users' sensitive data. We design two private RL algorithms that are based on value iteration and policy optimization, respectively, and show that they enjoy sub-linear regret performance while guaranteeing privacy protection. Moreover, the regret bounds are independent of the number of states, and scale at most logarithmically with the number of actions, making the algorithms suitable for privacy protection in nowadays large scale personalized services. Our results are achieved via a general procedure for learning in linear mixture MDPs under changing regularizers, which not only generalizes previous results for non-private learning, but also serves as a building block for general private reinforcement learning.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: On a Single User And For All Subset Of Actionsmentioning

confidence: 99%

See 1 more Smart Citation

Differentially Private Reinforcement Learning with Linear Function Approximation

Zhou¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Chen et al (2020) studied combinatorial bandits with LDP guarantees. Zheng et al (2020) studied both MABs and contextual bandits, and proposed a locally differentially private algorithm for contextual linear bandits. However, differentially private RL is much less studied compared with bandits, even though MDPs are more powerful since state transition is rather common in real applications.…”

Section: Introductionmentioning

confidence: 99%

“…T { pe ε pe ε ´1qqq lower bound for ε-LDP contextual linear bandits. This suggests that the algorithms proposed in Zheng et al (2020) might be improvable as well.…”

Section: Introductionmentioning

confidence: 99%

Locally Differentially Private Reinforcement Learning for Linear Mixture Markov Decision Processes

Liao¹,

He²,

Gu³

2021

Preprint

View full text Add to dashboard Cite

Reinforcement learning (RL) algorithms can be used to provide personalized services, which rely on users' private and sensitive data. To protect the users' privacy, privacy-preserving RL algorithms are in demand. In this paper, we study RL with linear function approximation and local differential privacy (LDP) guarantees. We propose a novel pε, δq-LDP algorithm for learning a class of Markov decision processes (MDPs) dubbed linear mixture MDPs, and obtains an1{4 a 1{εq regret, where d is the dimension of feature mapping, H is the length of the planning horizon, and T is the number of interactions with the environment. We also prove a lower bound ΩpdH ? T { pe ε pe ε ´1qqq for learning linear mixture MDPs under ε-LDP constraint. Experiments on synthetic datasets verify the effectiveness of our algorithm. To the best of our knowledge, this is the first provable privacy-preserving RL algorithm with linear function approximation.

show abstract

Distributed Linear Bandits With Differential Privacy

Li,

Zhou,

2024

IEEE Trans. Netw. Sci. Eng.

View full text Add to dashboard Cite

In this paper, we study the problem of global reward maximization with only partial distributed feedback. This problem is motivated by several real-world applications (e.g., cellular network configuration, dynamic pricing, and policy selection) where an action taken by a central entity influences a large population that contributes to the global reward. However, collecting such reward feedback from the entire population not only incurs a prohibitively high cost, but often leads to privacy concerns. To tackle this problem, we consider distributed linear bandits with differential privacy, where a subset of users from the population are selected (called clients) to participate in the learning process and the central server learns the global model from such partial feedback by iteratively aggregating these clients' local feedback in a differentially private fashion. We then propose a unified algorithmic learning framework, called differentially private distributed phased elimination (DP-DPE), which can be naturally integrated with popular differential privacy (DP) models (including central DP, local DP, and shuffle DP). Furthermore, we show that DP-DPE achieves both sublinear regret and sublinear communication cost. Interestingly, DP-DPE also achieves privacy protection "for free" in the sense that the additional cost due to privacy guarantees is a lower-order additive term. In addition, as a by-product of our techniques, the same results of "free" privacy can also be achieved for the standard differentially private linear bandits. Finally, we conduct simulations to corroborate our theoretical results and demonstrate the effectiveness of DP-DPE.

show abstract

Locally Differentially Private (Contextual) Bandits Learning

Cited by 10 publications

References 13 publications

Differentially Private Reinforcement Learning with Linear Function Approximation

Differentially Private Reinforcement Learning with Linear Function Approximation

Locally Differentially Private Reinforcement Learning for Linear Mixture Markov Decision Processes

Distributed Linear Bandits With Differential Privacy

Contact Info

Product

Resources

About