Differentially Private Simple Linear Regression

Abstract

Economics and social science research often require analyzing datasets of sensitive personal information at fine granularity, with models fit to small subsets of the data. Unfortunately, such fine-grained analysis can easily reveal sensitive individual information. We study regression algorithms that satisfy differential privacy, a constraint which guarantees that an algorithm’s output reveals little about any individual input data record, even to an attacker with side information about the dataset. Motivated by the Opportunity Atlas, a high-profile, small-area analysis tool in economics research, we perform a thorough experimental evaluation of differentially private algorithms for simple linear regression on small datasets with tens to hundreds of records—a particularly challenging regime for differential privacy. In contrast, prior work on differentially private linear regression focused on multivariate linear regression on large datasets or asymptotic analysis. Through a range of experiments, we identify key factors that affect the relative performance of the algorithms. We find that algorithms based on robust estimators—in particular, the median-based estimator of Theil and Sen—perform best on small datasets (e.g., hundreds of datapoints), while algorithms based on Ordinary Least Squares or Gradient Descent perform better for large datasets. However, we also discuss regimes in which this general finding does not hold. Notably, the differentially private analogues of Theil–Sen (one of which was suggested in a theoretical work of Dwork and Lei) have not been studied in any prior experimental work on differentially private linear regression.

Differentially Private Regression with Unbounded Covariates

“…Crucially, in contrast to the majority of prior works on regression, samples 𝑋 𝑖 are indeed unbounded, as they are sampled from 𝒩 (𝜇, Σ). Finally, the boundedness of the outputs 𝑦 𝑖 , 𝑖 ∈ [𝑛], is a requirement we share with other works (e.g., Alabi et al (2020); Wang (2018); Kifer et al (2012); Zhang et al (2012)), and clearly applies to, e.g., binary classification; we also study unbounded labels in the Linear Regression setting.…”

“…Linear regression is of course a true workhorse of statistics, and there has been a significant body of work on the design of computationally and statistically efficient differentially private regression algorithms (see e.g., the recent surveys of Cai et al (2020); Wang (2018) and the references therein). Approaches include objective perturbation (Iyengar et al, 2019;Kifer et al, 2012;Zhang et al, 2012;Chaudhuri et al, 2011), output perturbation (Asi and Duchi, 2020;Iyengar et al, 2019;Zhang et al, 2017;Jain and Thakurta, 2014), gradient perturbation (Abadi et al, 2016;Bassily et al, 2014), subsample-and-aggregate (Barrientos et al, 2019;Dwork and Smith, 2010), and sufficient statistics perturbation (Alabi et al, 2020;Wang, 2018;McSherry and Mironov, 2009). Additionally, several works study generalizations of such mechanisms to Generalized Linear Models (GLMs) (Kulkarni et al, 2021;Iyengar et al, 2019;Jain and Thakurta, 2014;Kifer et al, 2012).…”

Non-parametric Differentially Private Confidence Intervals for the Median

“…However, finding a differentially private estimator for this task that is accurate across a range of datasets and parameter regimes is surprisingly nuanced. There has been a significant amount of prior work on differentially private point estimators for the median [Nissim et al, 2007, Bun and Steinke, 2019, Asi and Duchi, 2020, Alabi et al, 2020, Tzamos et al, 2020 and other quantiles [Gillenwater et al, 2021]. To the best of our knowledge, none of these works addressed DP confidence intervals for the median.…”

“…Our first private mechanism is an instantiation of the exponential mechanism [McSherry and Talwar, 2007], a differentially private algorithm designed for general optimization problems. The exponential mechanism has been used in prior work to give DP point estimates for the median [Dwork and Lei, 2009, Thakurta and Smith, 2013, Johnson and Shmatikov, 2013, Alabi et al, 2020, Asi and Duchi, 2020. Our extension to providing confidence intervals for the median, while using similar ideas to prior work, requires a careful coverage analysis that is new to this work.…”

“…Definition B.1 (θ -Widened Exponential Mechanism [Alabi et al, 2020]). For a widening parameter θ > 0 and target rank k ∈ [1, 2, .…”