Differentially Private Query Release Through Adaptive Projection

Abstract

We propose, implement, and evaluate a new algorithm for releasing answers to very large numbers of statistical queries like k-way marginals, subject to differential privacy. Our algorithm makes adaptive use of a continuous relaxation of the Projection Mechanism, which answers queries on the private dataset using simple perturbation, and then attempts to find the synthetic dataset that most closely matches the noisy answers. We use a continuous relaxation of the synthetic dataset domain which makes the projection loss differentiable, and allows us to use efficient ML optimization techniques and tooling. Rather than answering all queries up front, we make judicious use of our privacy budget by iteratively and adaptively finding queries for which our (relaxed) synthetic data has high error, and then repeating the projection. We perform extensive experimental evaluations across a range of parameters and datasets, and find that our method outperforms existing algorithms in many cases, especially when the privacy budget is small or the query class is large.

Iterative Methods for Private Synthetic Data: Unifying Framework and New Methods

“…For example, we can recover existing methods by specifying choices of loss functions-we rederive MWEM [18] using an entropy-regularized linear loss, FEM [34] using a linear loss with a linear perturbation, and DualQuery [16] with a simple linear loss. Lastly, our framework lends itself naturally to a softmax variant of RAP [4], which we show outperforms the original RAP method itself.…”

“…While this body of work establishes optimal statistical rates for this problem, their proposed algorithms, including MWEM [19], typically have running time exponential in the dimension of the data. While the worst-case exponential running time is necessary (given known lower bounds [14,30,33]), a recent line of work on practical algorithms leverage optimization heuristics to tackle such computational bottlenecks [16,34,4]. In particular, DualQuery [16] and FEM [34] leverage integer program solvers to solve their NP-hard subroutines, and RAP [4] uses gradient-based methods to solve its projection step.…”

“…While the worst-case exponential running time is necessary (given known lower bounds [14,30,33]), a recent line of work on practical algorithms leverage optimization heuristics to tackle such computational bottlenecks [16,34,4]. In particular, DualQuery [16] and FEM [34] leverage integer program solvers to solve their NP-hard subroutines, and RAP [4] uses gradient-based methods to solve its projection step. In Section 3, we demonstrate how these algorithms can be viewed as special cases of our algorithmic framework.…”

“…RAP softmax adapted from Aydore et al [4] We note that RAP follows the basic structure of Adaptive Measurements, where at iteration t, RAP solves the following optimization problem:…”

J. Goseling and M.N.M. van Lieshout's Contribution to the Discussion of ‘Gaussian Differential Privacy’ by Donget al.

“…In particular, synthetic data is known to be neither necessary nor sufficient for privacy-but also not incompatible with differential privacy. For example, there is a large literature on generating differentially private synthetic data (see, e.g., Blum et al (2013); Gaboardi et al (2014); Vietri et al (2020); Aydore et al (2021); Jordon et al (2018); Beaulieu-Jones et al (2019)), most of which we believe can be improved by f-DP style analyses.…”