Differentially private empirical risk minimization for AUC maximization

Wang, Puyu; Yang, Zhenhuan; Lei, Yunwen; Ying, Yiming; Zhang, Hai

doi:10.1016/j.neucom.2021.07.001

Cited by 75 publications

(165 citation statements)

References 12 publications

Supporting

Mentioning

147

Contrasting

Order By: Relevance

“…In this regime, other algorithms (e.g. gradient descent output perturbation [ZZMW17], noisy SVRG [WYX17]) that use the alternative form of Gaussian noise are not even differentially private in general, as shown in [ZWB + 19, Theorem 1]. For ǫ ≤ 1, the condition √ ǫ c δ automatically holds for any δ ∈ 0, 1 2 .…”

Section: Excess Risk Bounds For Strongly Convex Lipschitz Functionsmentioning

confidence: 99%

See 1 more Smart Citation

Output Perturbation for Differentially Private Convex Optimization with Improved Population Loss Bounds, Runtimes and Applications to Private Adversarial Training

Lowy,

Razaviyayn

2021

Preprint

View full text Add to dashboard Cite

Finding efficient, easily implementable differentially private algorithms that offer strong excess risk bounds is an important problem in modern machine learning. To date, most work has focused on private empirical risk minimization (ERM) or private population loss minimization. However, there are often other objectives-such as fairness, adversarial robustness, or sensitivity to outliers-besides average performance that are not captured in the classical ERM setup. To this end, we study a completely general family of convex, Lipschitz loss functions and establish the first known differentially private excess risk and runtime bounds for optimizing this broad class. We provide similar bounds under additional assumptions of β-smoothness and/or strong convexity. We also address private stochastic convex optimization (SCO). While (ǫ, δ)-differential privacy (δ > 0) has been the focus of much recent work in private SCO, proving tight excess population loss bounds and runtime bounds for (ǫ, 0)-differential privacy remains a challenging open problem. We provide the tightest known (ǫ, 0)differentially private expected population loss bounds and fastest runtimes under the presence of (or lack of) smoothness and strong convexity assumptions. Our methods extend to the δ > 0 setting, where we offer the unique benefit of ensuring differential privacy for arbitrary ǫ > 0 by incorporating a new form of Gaussian noise proposed in [ZWB + 19]. Finally, we apply our theory to two learning frameworks: tilted ERM and adversarial learning. In particular, our theory quantifies tradeoffs between adversarial robustness, privacy, and runtime. Our results are achieved using perhaps the simplest yet practical differentially private algorithm: output perturbation. Although this method is not novel conceptually, our novel implementation scheme and analysis show that the power of this method to achieve strong privacy, utility, and runtime guarantees has not been fully appreciated in prior works.

show abstract

Section: Excess Risk Bounds For Strongly Convex Lipschitz Functionsmentioning

confidence: 99%

“…In the case of empirical risk minimization (ERM), where F (w, X) = 1 n n i=1 f (w, x i ), algorithms for maintaining differential privacy while solving Equation (1) are well studied [CMS11,BST14,ZZMW17,WYX17]. Here (and throughout) X = (x 1 • • • , x n ) is a data set with observations in some set X ⊆ R q .…”

mentioning

confidence: 99%

Output Perturbation for Differentially Private Convex Optimization with Improved Population Loss Bounds, Runtimes and Applications to Private Adversarial Training

Lowy,

Razaviyayn

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…However, getting the optimal bounds with small gradient complexity for non-smooth case turns out to be a more difficult problem. This was noted by [WYX17], who raised it as an important open problem. This question was answered in [BFTT19], who gave an algorithm with almost optimal excess empirical risk.…”

Section: Introductionmentioning

confidence: 96%

“…In the gradient perturbation approach, we add noise to the first order information using optimization algorithms such as Stochastic Gradient Descent (SGD). This approach was first proposed in [BST14] and was later extended by [TTZ14,WYX17], and has lead to the state-of-the-art theoretical bounds for DP-ERM. For an experimental comparison of various approaches to solving DP-ERM we refer the readers to [RBHT09, INS + 19].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Private Non-smooth Empirical Risk Minimization and Stochastic Convex Optimization in Subquadratic Steps

Kulkarni,

Lee,

Liu

2021

Preprint

View full text Add to dashboard Cite

We study the differentially private Empirical Risk Minimization (ERM) and Stochastic Convex Optimization (SCO) problems for non-smooth convex functions. We get a (nearly) optimal bound on the excess empirical risk and excess population loss with subquadratic gradient complexity. More precisely, our differentially private algorithm requires O( N 3/2 d 1/8 + N 2 d ) gradient queries for optimal excess empirical risk, which is achieved with the help of subsampling and smoothing the function via convolution. This is the first subquadratic algorithm for the nonsmooth case when d is super constant. As a direct application, using the iterative localization approach of Feldman et al. [FKT20], we achieve the optimal excess population loss for stochastic convex optimization problem, with O(min{N 5/4 d 1/8 , N 3/2 d 1/8 }) gradient queries. Our work makes progress towards resolving a question raised by Bassily et al. [BFGT20], giving first algorithms for private ERM and SCO with subquadratic steps. We note that independently Asi et al. [AFKT21] gave other algorithms for private ERM and SCO with subquadratic steps.

show abstract

Differentially private Riemannian optimization

Han,

Mishra,

Jawanpuria

et al. 2024

Mach Learn

View full text Add to dashboard Cite

In this paper, we study the differentially private empirical risk minimization problem where the parameter is constrained to a Riemannian manifold. We introduce a framework for performing differentially private Riemannian optimization by adding noise to the Riemannian gradient on the tangent space. The noise follows a Gaussian distribution intrinsically defined with respect to the Riemannian metric on the tangent space. We adapt the Gaussian mechanism from the Euclidean space to the tangent space compatible to such generalized Gaussian distribution. This approach presents a novel analysis as compared to directly adding noise on the manifold. We further prove privacy guarantees of the proposed differentially private Riemannian (stochastic) gradient descent using an extension of the moments accountant technique. Overall, we provide utility guarantees under geodesic (strongly) convex, general nonconvex objectives as well as under the Riemannian Polyak-Łojasiewicz condition. Empirical results illustrate the versatility and efficacy of the proposed framework in several applications.

show abstract

Differentially private empirical risk minimization for AUC maximization

Cited by 75 publications

References 12 publications

Output Perturbation for Differentially Private Convex Optimization with Improved Population Loss Bounds, Runtimes and Applications to Private Adversarial Training

Output Perturbation for Differentially Private Convex Optimization with Improved Population Loss Bounds, Runtimes and Applications to Private Adversarial Training

Private Non-smooth Empirical Risk Minimization and Stochastic Convex Optimization in Subquadratic Steps

Differentially private Riemannian optimization

Contact Info

Product

Resources

About