Private Stochastic Convex Optimization: Efficient Algorithms for Non-smooth Objectives

Arora, Raman; Marinov, Teodor V.; Ullah, Enayat

doi:10.48550/arxiv.2002.09609

Cited by 2 publications

(3 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The gradient descent-based output perturbation method of [ZZMW17] achieves with probability 1−γ, excess population loss of O RL 1/2 (βd) 1/4 (nǫγ) 1/2 , but they do not provide a guarantee for expected excess population loss. For δ > 0, [BFTGT19], [FKT20], and [AMU20] all nearly achieve the optimal (by [BFTGT19]) expected excess population loss bound O LR…”

Section: Smooth Convex Lipschitz Functionsmentioning

confidence: 99%

“…More recently, several works have also considered private stochastic convex optimization (SCO), where the goal is to minimize the expected population loss F (w, X) = E x∼D [f (w, x)], given access to n i.i.d. samples FKT20,AMU20]. However, the algorithms in these works are only differentially private for δ > 0, which, as discussed earlier, provides substantially weaker privacy guarantees.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Output Perturbation for Differentially Private Convex Optimization with Improved Population Loss Bounds, Runtimes and Applications to Private Adversarial Training

Lowy,

Razaviyayn

2021

Preprint

View full text Add to dashboard Cite

Finding efficient, easily implementable differentially private algorithms that offer strong excess risk bounds is an important problem in modern machine learning. To date, most work has focused on private empirical risk minimization (ERM) or private population loss minimization. However, there are often other objectives-such as fairness, adversarial robustness, or sensitivity to outliers-besides average performance that are not captured in the classical ERM setup. To this end, we study a completely general family of convex, Lipschitz loss functions and establish the first known differentially private excess risk and runtime bounds for optimizing this broad class. We provide similar bounds under additional assumptions of β-smoothness and/or strong convexity. We also address private stochastic convex optimization (SCO). While (ǫ, δ)-differential privacy (δ > 0) has been the focus of much recent work in private SCO, proving tight excess population loss bounds and runtime bounds for (ǫ, 0)-differential privacy remains a challenging open problem. We provide the tightest known (ǫ, 0)differentially private expected population loss bounds and fastest runtimes under the presence of (or lack of) smoothness and strong convexity assumptions. Our methods extend to the δ > 0 setting, where we offer the unique benefit of ensuring differential privacy for arbitrary ǫ > 0 by incorporating a new form of Gaussian noise proposed in [ZWB + 19]. Finally, we apply our theory to two learning frameworks: tilted ERM and adversarial learning. In particular, our theory quantifies tradeoffs between adversarial robustness, privacy, and runtime. Our results are achieved using perhaps the simplest yet practical differentially private algorithm: output perturbation. Although this method is not novel conceptually, our novel implementation scheme and analysis show that the power of this method to achieve strong privacy, utility, and runtime guarantees has not been fully appreciated in prior works.

show abstract

Section: Smooth Convex Lipschitz Functionsmentioning

confidence: 99%

mentioning

confidence: 99%

Output Perturbation for Differentially Private Convex Optimization with Improved Population Loss Bounds, Runtimes and Applications to Private Adversarial Training

Lowy,

Razaviyayn

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…However, the convergence rates of these differentially private (DP) versions of SGD suffer in most regularized optimization problems, particularly when working with non-smooth regularizers, such as the widely used ℓ 1 penalty for managing sparse, high-dimensional settings (Shamir and Zhang, 2013). Recently, DP variants of mirror descent and Frank-Wolfe Statistica Sinica: Newly accepted Paper (accepted author-version subject to English editing) algorithms have been developed to address this problem (Wang et al, 2018;Arora et al, 2020;Asi et al, 2021;Kulkarni et al, 2021). Additionally, Gopi et al (2022) proposed a modification to the exponential mechanism, which allows for the optimal empirical and population risk in private solving of non-smooth objective functions.…”

Section: Introductionmentioning

confidence: 99%

Differentially Private Regularized Stochastic Convex Optimization with Heavy-Tailed Data

Xie,

Pietrosanu,

Liu

et al. 2024

STAT SINICA

View full text Add to dashboard Cite

Existing privacy guarantees for convex optimization algorithms do not apply to heavy-tailed data with regularized estimation. This is a notable gap in the differential privacy (DP) literature, given the broad prevalence of non-Gaussian data and penalized optimization problems. In this work, we propose three (ϵ, δ)-DP methods for regularized convex optimization and derive bounds on their population excess risks in a framework that accommodates heavy-tailed data with fewer assumptions (relative to previous works). This work is the first to address DP in generic regularized convex optimization problems with heavy-tailed responses. Two of our methods augment a basic (ϵ, δ)-DP algorithm with robust procedures for privately estimating minibatch gradients. Our numerical analyses highlight the performance of our methods relative to data dimensionality, batch size, and privacy budget, and suggest settings where each approach is favorable.

show abstract

Private Stochastic Convex Optimization: Efficient Algorithms for Non-smooth Objectives

Cited by 2 publications

References 12 publications

Output Perturbation for Differentially Private Convex Optimization with Improved Population Loss Bounds, Runtimes and Applications to Private Adversarial Training

Output Perturbation for Differentially Private Convex Optimization with Improved Population Loss Bounds, Runtimes and Applications to Private Adversarial Training

Differentially Private Regularized Stochastic Convex Optimization with Heavy-Tailed Data

Contact Info

Product

Resources

About