Privacy-Preserved Distributed Learning With Zeroth-Order Optimization

Abstract

We develop a privacy-preserving distributed algorithm to minimize a regularized empirical risk function when the first-order information is not available and data is distributed over a multi-agent network. We employ a zeroth-order method to minimize the associated augmented Lagrangian function in the primal domain using the alternating direction method of multipliers (ADMM). We show that the proposed algorithm, named distributed zeroth-order ADMM (D-ZOA), has intrinsic privacy-preserving properties. Most existing privacy-preserving distributed optimization/estimation algorithms exploit some perturbation mechanism to preserve privacy, which comes at the cost of reduced accuracy. Contrarily, by analyzing the inherent randomness due to the use of a zeroth-order method, we show that D-ZOA is intrinsically endowed with (ϵ, δ)−differential privacy. In addition, we employ the moments accountant method to show that the total privacy leakage of D-ZOA grows sublinearly with the number of ADMM iterations. D-ZOA outperforms the existing differentially-private approaches in terms of accuracy while yielding similar privacy guarantee. We prove that D-ZOA reaches a neighborhood of the optimal solution whose size depends on the privacy parameter. The convergence analysis also reveals a practically important trade-off between privacy and accuracy. Simulation results verify the desirable privacypreserving properties of D-ZOA and its superiority over the stateof-the-art algorithms as well as its network-wide convergence.

Privacy Protection Based on Federated Learning

“…After using the FedAvg algorithm to handle data imbalance, the data is balanced, and privacy protection research can be conducted using the data of each participating user. There is a risk of data theft during the upload process [20][21][22] . To reduce data risk, enhance attack resistance and privacy protection, the study combines CNN and differential privacy to construct a DPAGD-CNN model with adaptive gradient descent.…”

Privacy-Preserving Distributed Online Stochastic Optimization With Time-Varying Distributions

“…Privacy-preserving distributed (stochastic) optimization method has recently been studied, including the inherent privacy protection method [39], quantization-enabled privacy protection method [40], and differential privacy method [41]- [47]. An important result that the convergence and differential privacy with a finite cumulative privacy budget ε for an infinite number of iterations hold simultaneously has been given for distributed optimization in [41], but this can not be directly used for distributed stochastic optimization.…”

“…Based on the gradient-perturbation mechanism [39] or a stochastic ternary quantization scheme [40], the privacy protection distributed stochastic optimization algorithm with only one iteration was proposed, respectively. Two common methods have been proposed for differential privacy distributed stochastic optimization, namely, gradient-perturbation [42]- [45] and output-perturbation [42], [46], [47]. However, the existing method induces a tradeoff between privacy and accuracy.…”

“…For the gradient-perturbation case, the mean square convergence of the proposed algorithm cannot be guaranteed, although a finite cumulative privacy budget ε for an infinite number of iterations has been presented in [43], [44], [45]. For the outputperturbation case, to guarantee the accuracy of the algorithm, ε-differential privacy was proven only for one iteration, leading to the cumulative privacy loss of kε after k iterations [42], [46], [47]. To the best of our knowledge, the convergence of the algorithm and differential privacy with a finite cumulative privacy budget ε for an infinite number of iterations has not been simultaneously established for distributed stochastic optimization.…”

“…Compared with [42], [43], [44], the mean-square and almost sure convergence of the algorithm can be guaranteed for both gradient-and output-perturbation methods. Compared with [40], [42]- [47], a finite cumulative privacy budget ε for an infinite number of iterations is proven for both gradient-and output-perturbation methods. ‚ The mean-square convergence rate of the algorithm with a two-time scale stochastic approximation-type step size is provided by properly selecting a Lyapunov function.…”