Privately Learning High-Dimensional Distributions

Abstract

We present novel, computationally efficient, and differentially private algorithms for two fundamental high-dimensional learning problems: learning a multivariate Gaussian and learning a product distribution over the Boolean hypercube in total variation distance. The sample complexity of our algorithms nearly matches the sample complexity of the optimal non-private learners for these tasks in a wide range of parameters, showing that privacy comes essentially for free for these problems. In particular, in contrast to previous approaches, our algorithm for learning Gaussians does not require strong a priori bounds on the range of the parameters. Our algorithms introduce a novel technical approach to reducing the sensitivity of the estimation procedure that we call recursive private preconditioning.

The Cost of Privacy: Optimal Rates of Convergence for Parameter Estimation with Differential Privacy

“…statistical setting. [21] studied Gaussian mean estimation but did not obtain a tight bound with respect to δ.…”

“…To demonstrate the abstract formulation in Theorem 2.1, we apply it to prove a concrete result, Theorem 2.2, that improved a similar lower bound in [21] by log(1/δ): when the sample size is n, any (ε, δ)-differentially private estimator of a d-dimensional sub-Gaussian mean vector must incur an extra error of at least the order of d log(1/δ)/nε, in addition to the standard d/n statistical error. The utility of Theorem 2.1 is further shown in Theorems 3.1, 4.1 and 4.3, for high-dimensional sparse mean estimation as well as linear regression in both low-dimensional and high-dimensional settings.…”

“…Early works in this direction were primarily concerned with the accuracy of releasing in-sample quantities, such as k-way marginals, with differential privacy constraints. Some more recent works [15,21] applied the idea to obtain lower bounds for discrete and Gaussian mean estimation. It is well known that theoretical properties of sample quantities can be very different from those of population parameters (see more discussions in [8]).…”

“…This tracing adversary argument, originally proposed by [4], has proven to be a powerful tool for obtaining lower bounds in the context of releasing sample quantities [32,33] and for Gaussian mean estimation [21]. We shall refine the tracing adversary argument to formulate a minimax lower bound technique that will be applied to a series of statistical estimation problems later in the paper.…”

Differentially Private High Dimensional Sparse Covariance Matrix Estimation

“…However, there are also evidences showing that the loss of some problems under the privacy constraints can be quite small compared with their non-private counterparts. Examples of such nature include high dimensional sparse PCA [8], sparse inverse covariance estimation [9], and high-dimensional distributions estimation [10]. Thus, it is desirable to determine which high dimensional problem can be learned or estimated efficiently in a private manner.…”