Private and polynomial time algorithms for learning Gaussians and beyond

Abstract

We present a fairly general framework for reducing (ε, δ) differentially private (DP) statistical estimation to its non-private counterpart. As the main application of this framework, we give a polynomial time and (ε, δ)-DP algorithm for learning (unrestricted) Gaussian distributions in R d . The sample complexity of our approach for learning the Gaussian up to total variation distance, matching (up to logarithmic factors) the best known informationtheoretic (non-efficient) sample complexity upper bound (Aden-Ali, Ashtiani, Kamath [1]). In an independent work, Kamath, Mouzakis, Singhal, Steinke, and Ullman [22] proved a similar result using a different approach and with O(d 5/2 ) sample complexity dependence on d.As another application of our framework, we provide the first polynomial time (ε, δ)-DP algorithm for robust learning of (unrestricted) Gaussians.

Fast, Sample-Efficient, Affine-Invariant Private Mean and Covariance Estimation for Subgaussian Distributions

“…Our accuracy analyses assume the underlying distribution has a full-rank covariance matrix. There exist sample-and time-efficient preprocessing algorithms to deal with distributions where this assumption fails ; ; Ashtiani and Liaw (2021)). Our privacy analysis makes no assumptions on the data.…”

“…For this task the sample complexity is well-understood, with lower bounds (Vadhan, 2017; and clean information-theoretic upper bounds (Aden-Ali et al, 2021) matching the non-private sample complexity for modest privacy parameters. A series of improving upper bounds has established polynomial-time algorithms nearly matching these results (Liu et al, 2022;Kothari et al, 2021;Ashtiani and Liaw, 2021;Alabi et al, 2022;Hopkins et al, 2022). For more comprehensive overviews (which also discuss robustness and stronger notions of privacy) refer to Alabi et al (2022) andHopkins et al (2022).…”

“…Our approach for stable mean estimation has a similar flavor to recently published tools in private high-dimensional statistics (Tsfadia et al (2022); Ashtiani and Liaw (2021)). Our work also shares a standard concentration argument with Tsfadia et al (2022), who also argue that checking the "proximity" to a small random sample of points preserves privacy while decreasing running time.…”

“…By contrast, we use stability of StableCovariance as a key ingredient of the proof of privacy for the mean estimation algorithm which adds noise scaled according to StableCovariance(x) to StableMean(x). This concept had been used elsewhere recently in various forms (Kothari et al (2021); Ashtiani and Liaw (2021); Tsfadia et al (2022); Alabi et al (2022)).…”

“…Our work (combining Algorithms 1 and 6) matches the optimal sample complexity up to logarithmic factors and runs in time Õ(nd ω−1 + nd/ε). To the best of our knowledge, the fastest known algorithm takes time Õ(nd ω−1 + d ω+1 /ε) and arises by combining ideas of Ashtiani and Liaw (2021) and Tsfadia et al (2022). For this task's typical parameter setting of n d 2 /ε, both of these expressions are asymptotically dominated by their first terms, corresponding to the time needed to compute the covariance of the entire input data.…”