Private Stochastic Non-Convex Optimization: Adaptive Algorithms and Tighter Generalization Bounds

Abstract

We study differentially private (DP) algorithms for stochastic non-convex optimization. In this problem, the goal is to minimize the population loss over a p-dimensional space given n i.i.d. samples drawn from a distribution. We improve upon the population gradient bound of √ p/ √ n from prior work and obtain a sharper rate of 4 √ p/ √ n. We obtain this rate by providing the first analyses on a collection of private gradient-based methods, including adaptive algorithms DP RMSProp and DP Adam. Our proof technique leverages the connection between differential privacy and adaptive data analysis to bound gradient estimation error at every iterate, which circumvents the worse generalization bound from the standard uniform convergence argument. Finally, we evaluate the proposed algorithms on two popular deep learning tasks and demonstrate the empirical advantages of DP adaptive gradient methods over standard DP SGD.

Differentially Private Learning Needs Better Features (or Much More Data)

“…suggests that faster convergence of non-private training could translate to better private learning. DP-SGD with adaptive updates (e.g.,Adam (Kingma & Ba, 2015)) indeed sometimes leads to small improvements(Papernot et al, 2020b;Chen & Lee, 2020;Zhou et al, 2020a). Investigating private variants of second-order optimization methods is an interesting direction for future work.• More training steps (a.k.a more data): For a fixed DP-budget ε and noise scale σ, increasing the training set size N allows for running more steps of DP-SGD(McMahan et al, 2018).…”

Adaptive Differentially Private Empirical Risk Minimization

“…The bounded gradient assumption is a common assumption for the analysis of DP-SGD algorithms , Zhou et al, 2020a and also frequently used in general adaptive gradient methods such as Adam [Reddi et al, 2021, Chen et al, 2018, Reddi et al, 2018. One recent popular approach to relax this assumption is using the gradient clipping method [Chen et al, 2020, Andrew et al, 2019, Pichapati et al, 2019, which we will discuss more in Section 6 as well as in Appendix A.…”

“…While most of them focus on convex functions, we study DP-ERM with nonconvex loss functions. As most existing algorithms achieving differential privacy in ERM are based on the gradient perturbation [Bassily et al, 2014, Zhou et al, 2020a, we thus study gradient perturbation.…”

“…where the loss function f (•) : R d × X → R is non-convex and smooth at each data point. To measure the performance of gradient-based algorithms for ERM, which enjoys privacy guarantees, we define the utility by using the expected 2 -norm of gradient, i.e., E[ ∇F (θ) ], where the expectation is taken over the randomness of the algorithm , Zhang et al, 2017, Zhou et al, 2020a]. 1 The DP-SGD (ADP-SGD), represented by the solid line.…”

“…Recent popular techniques for tuning η t include adaptive gradient methods Duchi et al [2011] and decaying stepsize schedules Goyal et al [2017]. When applying non-constant stepsizes, most of the existing differentially private algorithms directly follow the standard DP-SGD strategy by adding a simple perturbation (i.e, Z ∼ N (0, σ 2 I)) to each gradient over the entire sequence of iterations [Zhou et al, 2020a]. This results in a uniformly-distributed privacy budget for each iteration [Bassily et al, 2014].…”