Generation and evaluation of privacy preserving synthetic health data

Abstract

We develop metrics for measuring the quality of synthetic health data for both education and research. We use novel and existing metrics to capture a synthetic dataset's resemblance, privacy, utility and footprint. Using these metrics, we develop an end-to-end workflow based on our generative adversarial network (GAN) method, HealthGAN, that creates privacy preserving synthetic health data. Our workflow meets privacy specifications of our data partner: (1) the HealthGAN is trained inside a secure environment; (2) the HealthGAN model is used outside of the secure environment by external users to generate synthetic data. This second step facilitates data handling for external users by avoiding de-identification, which may require special user training, be costly, or cause loss of data fidelity. This workflow is compared against five other baseline methods. While maintaining resemblance and utility comparable to other methods, HealthGAN provides the best privacy and footprint. We present two case studies in which our methodology was put to work in the classroom and research settings. We evaluate utility in the classroom through a data analysis challenge given to students and in research by replicating three different medical papers with synthetic data. Data, code, and the challenge that we organized for educational purposes are available.

Synthetic Observational Health Data with GANs: from slow adoption to a boom in medical research and ultimately digital twins?

“…Jackson and Lussetti tested medGAN on an extended dataset containing demographic and health system usage information, obtaining results similar to the original (Jackson and Lussetti, 2019). HealthGAN, based on WGAN-GP, includes a data transformation method adapted from the Synthetic Data Vault (Patki et al, 2016) to map categorical features to and from the unit numerical range (Yale et al, 2020).…”

“…Dimensions-wise distribution The real and synthetic data are compared feature-wise according to a variety of methods For example, the Bernoulli success probability for binary features, or the Student T-test for continuous variables, and Pearson Chi-square test for binary variables is used to determine statistical significance (Beaulieu-Jones et al, 2019;Choi et al, 2017a;Chin-Cheong et al, 2019;Baowaly et al, 2019;Baowaly et al, 2018;Ozyigit et al, 2020;Tantipongpipat et al, 2019;Tantipongpipat et al, 2019;Fisher et al, 2019;Wang et al, 2019a;Yale et al, 2019a;Chin-Cheong et al, 2020;Ozyigit et al, 2020).…”

“…Inter-dimensional correlation Dimension-wise Pearson coefficient correlation matrices for both real and synthetic data (Beaulieu-Jones et al, 2019;Goncalves et al, 2020;Torfi and Beyki, 2019;Frid-Adar et al, 2018;Ozyigit et al, 2020;Yang et al, 2019c;Zhu et al, 2020a;Walsh et al, 2020;Yale et al, 2019a;Ozyigit et al, 2020;Dash et al, 2019;Bae et al, 2020b).…”

“…A predictor is trained to discriminate synthetic from real sample. An accuracy value of 0.5 would indicate that they are indistinguishable (Fisher et al, 2019;Walsh et al, 2020;Yale et al, 2019b).…”

“…For example, they organized a hack-a-thon type challenge involving the data. During the event, students were tasked with creating classifiers, while provided only with SD (Yale et al, 2020). They were then scored on the accuracy of their model on real data.…”