Distributed Learning with Non-Smooth Objective Functions

Gratton, Cristiano; Venkategowda, Naveen K. D.; Arablouei, Reza; Werner, Stefan

doi:10.23919/eusipco47968.2020.9287441

Cited by 3 publications

(3 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…where α 0 is an appropriate initial step-size and R is an upper bound on the distance between a minimizer w * to (7) and the first iterate w (1) as per [25].…”

Section: Zeroth-order Methodsmentioning

confidence: 99%

Privacy-Preserving Distributed Zeroth-Order Optimization

Gratton¹,

Venkategowda²,

Arablouei³

et al. 2020

Preprint

View full text Add to dashboard Cite

We develop a privacy-preserving distributed algorithm to minimize a regularized empirical risk function when the first-order information is not available and data is distributed over a multi-agent network. We employ a zeroth-order method to minimize the associated augmented Lagrangian function in the primal domain using the alternating direction method of multipliers (ADMM). We show that the proposed algorithm, named distributed zeroth-order ADMM (D-ZOA), has intrinsic privacy-preserving properties. Unlike the existing privacypreserving methods based on the ADMM where the primal or the dual variables are perturbed with noise, the inherent randomness due to the use of a zeroth-order method endows D-ZOA with intrinsic differential privacy. By analyzing the perturbation of the primal variable, we show that the privacy leakage of the proposed D-ZOA algorithm is bounded. In addition, we employ the moments accountant method to show that the total privacy leakage grows sublinearly with the number of ADMM iterations. D-ZOA outperforms the existing differentially private approaches in terms of accuracy while yielding the same privacy guarantee. We prove that D-ZOA converges to the optimal solution at a rate of O(1/M ) where M is the number of ADMM iterations. The convergence analysis also reveals a practically important tradeoff between privacy and accuracy. Simulation results verify the desirable privacy-preserving properties of D-ZOA and its superiority over a state-of-the-art algorithm as well as its network-wide convergence to the optimal solution.

show abstract

“…where α 0 is an appropriate initial step-size and R is an upper bound on the distance between a minimizer w * to (7) and the first iterate w (1) as per [25].…”

Section: Zeroth-order Methodsmentioning

confidence: 99%

Privacy-Preserving Distributed Zeroth-Order Optimization

Gratton¹,

Venkategowda²,

Arablouei³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Since the objective function in ( 4) is assumed to be nonsmooth, the corresponding minimization problem cannot be solved using any first-order method. To overcome this, we use a zeroth-order method as in [1]. We utilize the twopoint stochastic-gradient algorithm that has been proposed in [29] for optimizing general non-smooth functions.…”

Section: B Zeroth-order-based Distributed Admm Algorithmmentioning

confidence: 99%

“…We utilize the twopoint stochastic-gradient algorithm that has been proposed in [29] for optimizing general non-smooth functions. More specifically, we use the stochastic mirror descent method with the proximal function 1 2 ∥•∥ and the gradient estimator at point β k given by…”

Section: B Zeroth-order-based Distributed Admm Algorithmmentioning

confidence: 99%

Privacy-Preserved Distributed Learning With Zeroth-Order Optimization

Gratton

Venkategowda

Arablouei

et al. 2022

IEEE Trans.Inform.Forensic Secur.

Self Cite

View full text Add to dashboard Cite

We develop a privacy-preserving distributed algorithm to minimize a regularized empirical risk function when the first-order information is not available and data is distributed over a multi-agent network. We employ a zeroth-order method to minimize the associated augmented Lagrangian function in the primal domain using the alternating direction method of multipliers (ADMM). We show that the proposed algorithm, named distributed zeroth-order ADMM (D-ZOA), has intrinsic privacy-preserving properties. Most existing privacy-preserving distributed optimization/estimation algorithms exploit some perturbation mechanism to preserve privacy, which comes at the cost of reduced accuracy. Contrarily, by analyzing the inherent randomness due to the use of a zeroth-order method, we show that D-ZOA is intrinsically endowed with (ϵ, δ)−differential privacy. In addition, we employ the moments accountant method to show that the total privacy leakage of D-ZOA grows sublinearly with the number of ADMM iterations. D-ZOA outperforms the existing differentially-private approaches in terms of accuracy while yielding similar privacy guarantee. We prove that D-ZOA reaches a neighborhood of the optimal solution whose size depends on the privacy parameter. The convergence analysis also reveals a practically important trade-off between privacy and accuracy. Simulation results verify the desirable privacypreserving properties of D-ZOA and its superiority over the stateof-the-art algorithms as well as its network-wide convergence.

show abstract

Distributed Learning over Networks with Non-Smooth Regularizers and Feature Partitioning

Gratton

Venkategowda

Arablouei

et al. 2021

2021 29th European Signal Processing Conference (EUSIPCO)

Self Cite

View full text Add to dashboard Cite

We develop a new algorithm for distributed learning with non-smooth regularizers and feature partitioning. To this end, we transform the underlying optimization problem into a suitable dual form and solve it using the alternating direction method of multipliers. The proposed algorithm is fullydistributed and does not require the conjugate function of any non-smooth regularizer function, which may be unfeasible or computationally inefficient to acquire. Numerical experiments demonstrate the effectiveness of the proposed algorithm.

show abstract

Distributed Learning with Non-Smooth Objective Functions

Cited by 3 publications

References 24 publications

Privacy-Preserving Distributed Zeroth-Order Optimization

Privacy-Preserving Distributed Zeroth-Order Optimization

Privacy-Preserved Distributed Learning With Zeroth-Order Optimization

Distributed Learning over Networks with Non-Smooth Regularizers and Feature Partitioning

Contact Info

Product

Resources

About