On the Theory and Practice of Privacy-Preserving Bayesian Data Analysis

Abstract

Bayesian inference has great promise for the privacy-preserving analysis of sensitive data, as posterior sampling automatically preserves differential privacy, an algorithmic notion of data privacy, under certain conditions (Dimitrakakis et al., 2014;Wang et al., 2015b). While this one posterior sample (OPS) approach elegantly provides privacy "for free," it is data inefficient in the sense of asymptotic relative efficiency (ARE). We show that a simple alternative based on the Laplace mechanism, the workhorse of differential privacy, is as asymptotically efficient as non-private posterior inference, under general assumptions. This technique also has practical advantages including efficient use of the privacy budget for MCMC. We demonstrate the practicality of our approach on a time-series analysis of sensitive military records from the Afghanistan and Iraq wars disclosed by the Wikileaks organization.

Latent Dirichlet Allocation Model Training with Differential Privacy

“…For example, Wang et al [36] and Dimitrakakis et al [38] first demonstrated that, when meeting certain conditions of the loss function, the posterior Bayesian sampling and the stochastic gradient Markov Chain Monte Carlo (MCMC) techniques could possess some inherent privacy guarantee without the introduction of extra noise. Foulds et al [40] further extended this conclusion to the general MCMC methods. By utilizing the inherent randomness of MCMC, they achieved a certain level of privacy protection equivalent to that assured by a Laplace mechanism.…”

“…A direct method to achieve DP in the CGS-based LDA algorithm is to add noise to the inner statistics [40][17], e.g., word counts n t k and n k m , based on which the sampling probability would be computed to perform topic sampling. In [40], the authors provide a general method to achieve DP in Gibbs Sampling, i.e., adding Laplace noise to the sufficient statistics, n t k and n k m in LDA at the beginning of the Gibbs Sampling process. [17] proposes to add Laplace noise to n t k and n k m in the final iteration.…”

Privacy for Spatial Point Process Data

“…Assume we model the locations according to a LGCP with intensity λ(•) given by ( 2) with a spatial random field defined as in (4). As shown in Foulds et al [10], releasing one sample from the posterior distribution π(λ|S) is 2C-differentially private for any prior, provided…”

“…To see this note that (10) can equivalently be expressed as λ † (s) = exp (ν(s)) λ(s), where λ(s) denotes the intensity surface with plug-in posterior mean estimates β and ŵ. We also note that ANS can be viewed as an extension of the synthesis method proposed by Quick et al [17], in which synthetic locations were generated based on posterior mean plug-in estimates.…”

“…Dimitrakakis at al. [6] and Foulds et al [10] observed that Bayesian posterior sampling providesdifferential privacy under certain prior assumptions. Theorem 2 of Foulds et al [10] states that releasing one sample from the posterior distribution, π(x|θ), with any prior, π(θ), is 2C-differentially private provided…”