In this paper, we study private optimization problems for non-smooth convex functionsWe show that modifying the exponential mechanism by adding an ℓ 2 2 regularizer to F (x) and sampling from π(x) ∝ exp(−k(F (x) + µ x 2 2 /2)) recovers both the known optimal empirical risk and population loss under (ε, δ)-DP. Furthermore, we show how to implement this mechanism using O(n min(d, n)) queries to f i (x) for the DP-SCO where n is the number of samples/users and d is the ambient dimension. We also give a (nearly) matching lower bound Ω(n min(d, n)) on the number of evaluation queries.Our results utilize the following tools that are of independent interest:

No abstract

The development of efficient sampling algorithms catering to non-Euclidean geometries has been a challenging endeavor, as discretization techniques which succeed in the Euclidean setting do not readily carry over to more general settings. We develop a non-Euclidean analog of the recent proximal sampler of [LST21b], which naturally induces regularization by an object known as the log-Laplace transform (LLT) of a density. We prove new mathematical properties (with an algorithmic flavor) of the LLT, such as strong convexity-smoothness duality and an isoperimetric inequality, which are used to prove a mixing time on our proximal sampler matching [LST21b] under a warm start. As our main application, we show our warm-started sampler improves the value oracle complexity of differentially private convex optimization in ℓ p and Schatten-p norms for p ∈ [1, 2] to match the Euclidean setting [GLL22], while retaining state-of-the-art excess risk bounds [GLL + 23]. We find our investigation of the LLT to be a promising proof-of-concept of its utility as a tool for designing samplers, and outline directions for future exploration.

We propose a new framework for differentially private optimization of convex functions which are Lipschitz in an arbitrary norm • X . Our algorithms are based on a regularized exponential mechanism which samples from the density ∝ exp(−k(F + µr)) where F is the empirical loss and r is a regularizer which is strongly convex with respect to • X , generalizing a recent work of [GLL22] to non-Euclidean settings. We show that this mechanism satisfies Gaussian differential privacy and solves both DP-ERM (empirical risk minimization) and DP-SCO (stochastic convex optimization), by using localization tools from convex geometry. Our framework is the first to apply to private convex optimization in general normed spaces, and directly recovers non-private SCO rates achieved by mirror descent, as the privacy parameter ǫ → ∞. As applications, for Lipschitz optimization in ℓ p norms for all p ∈ (1, 2), we obtain the first optimal privacy-utility tradeoffs; for p = 1, we improve tradeoffs obtained by the recent works [AFKT21, BGN21] by at least a logarithmic factor. Our ℓ p norm and Schatten-p norm optimization frameworks are complemented with polynomial-time samplers whose query complexity we explicitly bound.