Recycled ADMM: Improve Privacy and Accuracy with Less Computation in Distributed Algorithms

Abstract

Alternating direction method of multiplier (ADMM) is a powerful method to solve decentralized convex optimization problems. In distributed settings, each node performs computation with its local data and the local results are exchanged among neighboring nodes in an iterative fashion. During this iterative process the leakage of data privacy arises and can accumulate significantly over many iterations, making it difficult to balance the privacy-utility tradeoff. In this study we propose Recycled ADMM (R-ADMM), where a linear approximation is applied to every even iteration, its solution directly calculated using only results from the previous, odd iteration. It turns out that under such a scheme, half of the updates incur no privacy loss and require much less computation compared to the conventional ADMM. We obtain a sufficient condition for the convergence of R-ADMM and provide the privacy analysis based on objective perturbation.

Differentially Private Robust ADMM for Distributed Machine Learning

“…In this section, we will present our plausible differentially private (PP-ADMM) by adding Gaussian noise related to the maximum tolerable gradient norm of perturbed objective in each ADMM iteration, which relaxes the requirement of exact optimal solution as shown in [23,25,26], to provide differential privacy guarantee of each training data sample during the iterative procedure. We also adopt the privacy framework of zCDP to compute much tighter privacy loss estimation of PP-ADMM.…”

“…Specifically, in each iteration, we perturb the subproblem (3) with the objective perturbation method the same as used in previous studies [23,25,26], where a random linear vector (b i1 ) T θ i is injected to the objective function, and b i1 is a random vector drawn from a zero mean Gaussian distribution N (0, σ 2 i1 I d ). Consequently the objective function (3) used to update the primal variable θ t +1 i becomes the following modified function:…”

“…In order to ensure DP guarantee, as pointed out in [23,25,26], each agent i ∈ N needs to find the optimal solutionθ t +1…”

“…Specifically, as an objective perturbation method, [23] proposed to inject noise to the update of the dual variable to provide DP guarantee, while the total privacy loss over the whole iterative process is not quantified. Further, [25,26] proposed to perturb the penalty parameter of ADMM and re-utilize the previous iteration's results to save the privacy loss. These methods also quantify the total privacy loss over the entire process.…”

Recycled ADMM: Improving the Privacy and Accuracy of Distributed Algorithms

“…This work is supported by the NSF under grants CNS-1422211, CNS-1646019, and CNS-1739517. An earlier version of this paper appeared in the 2018 Allerton Conference on Communication, Control and Computing [1]. X. Zhang, M. Khalili , where k is the number of iterations [12].…”