The Cost of Privacy: Optimal Rates of Convergence for Parameter Estimation with Differential Privacy

Abstract

Privacy-preserving data analysis is a rising challenge in contemporary statistics, as the privacy guarantees of statistical methods are often achieved at the expense of accuracy. In this paper, we investigate the tradeoff between statistical accuracy and privacy in mean estimation and linear regression, under both the classical lowdimensional and modern high-dimensional settings. A primary focus is to establish minimax optimality for statistical estimation with the (ε, δ)-differential privacy constraint. To this end, we find that classical lower bound arguments fail to yield sharp results, and new technical tools are called for.Inspired by the theoretical computer science literature on "tracing adversaries", we formulate a general lower bound argument for minimax risks with differential privacy constraints, and apply this argument to high-dimensional mean estimation and linear regression problems. We also design computationally efficient algorithms that attain the minimax lower bounds up to a logarithmic factor. In particular, for the high-dimensional linear regression, a novel private iterative hard thresholding pursuit algorithm is proposed, based on a privately truncated version of stochastic gradient descent. The numerical performance of these algorithms is demonstrated by simulation studies and applications to real data containing sensitive information, for which privacy-preserving statistical methods are necessary.

Differential privacy and robust statistics in high dimensions

“…Even for DP linear regression without robustness, HPTR is the first algorithm for sub-Gaussian distributions with an unknown covariance Σ that up to log factors matches the lower bound of n = Ω(d/α 2 + d/(αε)) assuming ε < 1 and δ < n −1−ω for some ω > 0 from [CWZ19, Theorem 4.1]. For completeness, we provide the lower bound in Appendix C. An existing algorithm for DP linear regression from [CWZ19] is suboptimal as it require Σ to be close to the identity matrix, which is equivalent to assuming that we know Σ.…”

“…There exists a (ε, δ)-differentially private algorithm β(S) that given HPTR is the first algorithm for sub-Gaussian distributions with an unknown covariance Σ that up to logarithmic factors matches the lower bound of n = Ω(d/ξ 2 + d/(ξε)) assuming ε < 1 and δ < n −1−ω for some ω > 0 from [CWZ19, Theorem 4.1]. An existing algorithm for DP linear regression from [CWZ19] is suboptimal as it require Σ to be close to the identity matrix, which is equivalent to assuming that we know Σ. [DL09] proposes to use PTR and B-robust regression algorithm from [HRRS86] for differentially private linear regression under i.i.d.…”

High-Dimensional Differentially-Private EM Algorithm: Methods and Near-Optimal Statistical Guarantees

“…The trade-off between statistical accuracy and privacy is one of the fundamental topics in differential privacy. In the low-dimensional setting, there are various works focusing on this trade-off, including mean estimation [28,52,8,34,10,36], confidence intervals of Gaussian mean [37] and binomial mean [4], linear regression [55,10], generalized linear models [51,11,50], principal component analysis [30,12], convex empirical risk minimization [7], and robust M-estimators [2,3].…”

“…Most of the existing works focus on relatively standard statistical problems such as the sparse mean estimation and regression. For example, [10] studies near-optimal algorithms for the sparse mean estimation.…”

“…For the high-dimensional sparse linear regression, [53] obtains a private LASSO algorithm which is near-optimal in terms of the excess risk; [10] proposes another private LASSO algorithm with the optimal rate of convergence in estimation. In [11], a near-minimax optimal differentially private algorithm for high-dimensional generalized linear models is introduced.…”