Near Instance-Optimality in Differential Privacy

Abstract

We develop two notions of instance optimality in differential privacy, inspired by classical statistical theory: one by defining a local minimax risk and the other by considering unbiased mechanisms and analogizing the Cramér-Rao bound, and we show that the local modulus of continuity of the estimand of interest completely determines these quantities. We also develop a complementary collection mechanisms, which we term the inverse sensitivity mechanisms, which are instance optimal (or nearly instance optimal) for a large class of estimands. Moreover, these mechanisms uniformly outperform the smooth sensitivity framework-on each instance-for several function classes of interest, including R-valued continuous functions. We carefully present two instantiations of the mechanisms for median and robust regression estimation with corresponding experiments.

Adapting to Function Difficulty and Growth Conditions in Private Optimization

“…Our goal is to design private optimization algorithms that adapt to the difficulty of the underlying function. As a reference point, we turn to the inverse sensitivity mechanism of [2] as it enjoys general instance-optimality guarantees. For a given function h : S n → T ⊂ R d that we wish to estimate privately, define the inverse sensitivity at x ∈ T…”

“…The inverse sensitivity mechanism preserves ε-DP and enjoys instance-optimality guarantees in general settings [2]. In contrast to (worst-case) minimax optimality guarantees which measure the performance of the algorithm on the hardest instance, these notions of instance-optimality provide stronger per-instance optimality guarantees.…”

“…As a simple vignette, if one gets an arbitrary 1-Lipschitz convex loss function f , the worst-case guarantee of any ε-DP algorithm is Θ(1/ √ n + d/(nε)). However, if one learns that f exhibits some growth property-say f is 1-strongly convex-the regret guarantee improves to the faster Θ(1/n + (d/(nε)) 2 ) rate with the appropriate algorithm. It is thus important to provide algorithms that achieves the rates of the "easiest" class to which the function belongs [32,46,18].…”

“…In this work, we provide private adaptive algorithms that adapt to the actual growth of the function at hand. We begin our analysis by examining Asi and Duchi's inverse sensitivity mechanism [2] on ERM as a motivation. While not a practical algorithm, it achieves instance-optimal rates for any onedimensional function under mild assumptions, quantifying the best bound one could hope to achieve with an adaptive algorithm, and showing (in principle) that adaptive private algorithms can exist.…”

“…Crucially, they achieve these rates without knowledge of the growth constant κ of the function. Our algorithms build upon the inverse sensitivity mechanism, which adapts to instance difficulty [2], and recent localization techniques in private optimization [25]. We complement our algorithms with matching lower bounds for these function classes and demonstrate that our adaptive algorithm is simultaneously (minimax) optimal over all κ ≥ 1 + c whenever c = Θ(1).…”