Stochastic ADMM Based Distributed Machine Learning with Differential Privacy

Ding, Jiahao; Errapotu, Sai Mounika; Zhang, Haijun; Gong, Yandao; Pan, Miao; Han, Zhu

doi:10.1007/978-3-030-37228-6_13

Cited by 20 publications

(15 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The above inequality holds for all γ i , thus it also holds for γ i ∈ {γ i : γ i ≤ β}. By letting γ i be the optimum, we have , which leads 19 to the result:…”

mentioning

confidence: 90%

“…Differential privacy is a widely used privacy definition [14]- [16] and can be guaranteed in ADMM through adding noise to the exchanged messages. However, in existing studies on ADMMbased distributed learning with differential privacy [1], [2], [17]- [19], noise addition would disrupt the learning process and severely degrade the performance of the trained model, especially when large noise is needed to provide high privacy protection. Besides, their privacy-preserving algorithms only apply to the learning problems with both smoothness and strongly convexity assumptions about the objective functions.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

DP-ADMM: ADMM-Based Distributed Learning With Differential Privacy

Huang

Guo

et al. 2020

IEEE Trans.Inform.Forensic Secur.

Self Cite

166

176

View full text Add to dashboard Cite

Alternating direction method of multipliers (ADMM) is a widely used tool for machine learning in distributed settings, where a machine learning model is trained over distributed data sources through an interactive process of local computation and message passing. Such an iterative process could cause privacy concerns of data owners. The goal of this paper is to provide differential privacy for ADMM-based distributed machine learning. Prior approaches on differentially private ADMM exhibit low utility under high privacy guarantee and assume the objective functions of the learning problems to be smooth and strongly convex. To address these concerns, we propose a novel differentially private ADMM-based distributed learning algorithm called DP-ADMM, which combines an approximate augmented Lagrangian function with time-varying Gaussian noise addition in the iterative process to achieve higher utility for general objective functions under the same differential privacy guarantee. We also apply the moments accountant method to analyze the end-to-end privacy loss. The theoretical analysis shows that DP-ADMM can be applied to a wider class of distributed learning problems, is provably convergent, and offers an explicit utility-privacy tradeoff. To our knowledge, this is the first paper to provide explicit convergence and utility properties for differentially private ADMM-based distributed learning algorithms. The evaluation results demonstrate that our approach can achieve good convergence and model accuracy under high end-to-end differential privacy guarantee.

show abstract

“…The above inequality holds for all γ i , thus it also holds for γ i ∈ {γ i : γ i ≤ β}. By letting γ i be the optimum, we have , which leads 19 to the result:…”

mentioning

confidence: 90%

Section: Introductionmentioning

confidence: 99%

DP-ADMM: ADMM-Based Distributed Learning With Differential Privacy

Huang

Guo

et al. 2020

IEEE Trans.Inform.Forensic Secur.

Self Cite

166

176

View full text Add to dashboard Cite

show abstract

“…35,36 Gao and Ma 37 proposed an algorithm which combines reinforcement with differential privacy in processing dynamic data. Ding et al 38 put forward a alternating direction method based on differential privacy and reinforcement. Cheng et al 39 proposed a novel stochastic gradient descent algorithm with deep learning and differential privacy.…”

Section: Differential Privacymentioning

confidence: 99%

A randomized block policy gradient algorithm with differential privacy in Content Centric Networks

Wang

Zhao

et al. 2021

International Journal of Distributed Sensor Networks

View full text Add to dashboard Cite

Policy gradient methods are effective means to solve the problems of mobile multimedia data transmission in Content Centric Networks. Current policy gradient algorithms impose high computational cost in processing high-dimensional data. Meanwhile, the issue of privacy disclosure has not been taken into account. However, privacy protection is important in data training. Therefore, we propose a randomized block policy gradient algorithm with differential privacy. In order to reduce computational complexity when processing high-dimensional data, we randomly select a block coordinate to update the gradients at each round. To solve the privacy protection problem, we add a differential privacy protection mechanism to the algorithm, and we prove that it preserves the [Formula: see text]-privacy level. We conduct extensive simulations in four environments, which are CartPole, Walker, HalfCheetah, and Hopper. Compared with the methods such as important-sampling momentum-based policy gradient, Hessian-Aided momentum-based policy gradient, REINFORCE, the experimental results of our algorithm show a faster convergence rate than others in the same environment.

show abstract

“…However, in both algorithms, the privacy leakage of an agent is bounded only at a single iteration and an adversary might exploit knowledge available from all iterations to infer sensitive information. This shortcoming is mitigated in [15]- [18]. The works in [15], [16] develop ADMM-based differentially private algorithms with improved accuracy.…”

Section: A Related Workmentioning

confidence: 99%

“…The work in [17] employs the ADMM to develop a distributed algorithm where the primal variable is perturbed by adding a Gaussian noise with diminishing variance to ensure zero-concentrated differential privacy enabling higher accuracy compared to the common (ǫ, δ)-differential privacy. The work in [18] develops a stochastic ADMM-based distributed algorithm that further enhances the accuracy while ensuring differential privacy. The authors of [19]- [21] propose differentially-private distributed algorithms that utilize the projected-gradient-descent method for handling constraints.…”

Section: A Related Workmentioning

confidence: 99%

Privacy-Preserving Distributed Zeroth-Order Optimization

Gratton¹,

Venkategowda²,

Arablouei³

et al. 2020

Preprint

View full text Add to dashboard Cite

We develop a privacy-preserving distributed algorithm to minimize a regularized empirical risk function when the first-order information is not available and data is distributed over a multi-agent network. We employ a zeroth-order method to minimize the associated augmented Lagrangian function in the primal domain using the alternating direction method of multipliers (ADMM). We show that the proposed algorithm, named distributed zeroth-order ADMM (D-ZOA), has intrinsic privacy-preserving properties. Unlike the existing privacypreserving methods based on the ADMM where the primal or the dual variables are perturbed with noise, the inherent randomness due to the use of a zeroth-order method endows D-ZOA with intrinsic differential privacy. By analyzing the perturbation of the primal variable, we show that the privacy leakage of the proposed D-ZOA algorithm is bounded. In addition, we employ the moments accountant method to show that the total privacy leakage grows sublinearly with the number of ADMM iterations. D-ZOA outperforms the existing differentially private approaches in terms of accuracy while yielding the same privacy guarantee. We prove that D-ZOA converges to the optimal solution at a rate of O(1/M ) where M is the number of ADMM iterations. The convergence analysis also reveals a practically important tradeoff between privacy and accuracy. Simulation results verify the desirable privacy-preserving properties of D-ZOA and its superiority over a state-of-the-art algorithm as well as its network-wide convergence to the optimal solution.

show abstract

Stochastic ADMM Based Distributed Machine Learning with Differential Privacy

Cited by 20 publications

References 14 publications

DP-ADMM: ADMM-Based Distributed Learning With Differential Privacy

DP-ADMM: ADMM-Based Distributed Learning With Differential Privacy

A randomized block policy gradient algorithm with differential privacy in Content Centric Networks

Privacy-Preserving Distributed Zeroth-Order Optimization

Contact Info

Product

Resources

About