Reliability of Supervised Machine Learning Using Synthetic Data in Health Care: Model to Preserve Privacy for Data Sharing

Abstract

Background The exploitation of synthetic data in health care is at an early stage. Synthetic data could unlock the potential within health care datasets that are too sensitive for release. Several synthetic data generators have been developed to date; however, studies evaluating their efficacy and generalizability are scarce. Objective This work sets out to understand the difference in performance of supervised machine learning models trained on synthetic data compared with those trained on real data. Methods A total of 19 open health datasets were selected for experimental work. Synthetic data were generated using three synthetic data generators that apply classification and regression trees, parametric, and Bayesian network approaches. Real and synthetic data were used (separately) to train five supervised machine learning models: stochastic gradient descent, decision tree, k-nearest neighbors, random forest, and support vector machine. Models were tested only on real data to determine whether a model developed by training on synthetic data can used to accurately classify new, real examples. The impact of statistical disclosure control on model performance was also assessed. Results A total of 92% of models trained on synthetic data have lower accuracy than those trained on real data. Tree-based models trained on synthetic data have deviations in accuracy from models trained on real data of 0.177 (18%) to 0.193 (19%), while other models have lower deviations of 0.058 (6%) to 0.072 (7%). The winning classifier when trained and tested on real data versus models trained on synthetic data and tested on real data is the same in 26% (5/19) of cases for classification and regression tree and parametric synthetic data and in 21% (4/19) of cases for Bayesian network-generated synthetic data. Tree-based models perform best with real data and are the winning classifier in 95% (18/19) of cases. This is not the case for models trained on synthetic data. When tree-based models are not considered, the winning classifier for real and synthetic data is matched in 74% (14/19), 53% (10/19), and 68% (13/19) of cases for classification and regression tree, parametric, and Bayesian network synthetic data, respectively. Statistical disclosure control methods did not have a notable impact on data utility. Conclusions The results of this study are promising with small decreases in accuracy observed in models trained with synthetic data compared with models trained with real data, where both are tested on real data. Such deviations are expected and manageable. Tree-based classifiers have some sensitivity to synthetic data, and the underlying cause requires further investigation. This study highlights the potential of synthetic data and the need for further evaluation of their robustness. Synthetic data must ensure individual privacy and data utility are preserved in order to instill confidence in health care departments when using such data to inform policy decision-making.

Meaningful Big Data Integration for a Global COVID-19 Strategy

“…Moreover, the combination of the MIDAS developed GYDRA data preparation tool, alongside synthetic dataset generation strategies, can enable hospitals and healthcare providers, to: 1) refine and prepare their datasets (with the required metadata description), and; 2) share synthetically generated privacypreserving datasets with the scientific community, that follow statistical patterns similar to the real data, and have proven to be reliable for training machine learning models [28]. These mechanisms would enable users to load a controlled dataset into the MIDAS platform and to develop in-house analytics, whilst simultaneously allowing the scientific community to develop AI models based on synthetic datasets that can later be fed back to the policy-makers through the MIDAS platform.…”

Synthetic Observational Health Data with GANs: from slow adoption to a boom in medical research and ultimately digital twins?

“…Such a knowledge-based model depends on prior knowledge of the system, and how much we can intellect about it (Kim et al, 2017;Bonnéry et al, 2019). On one hand, theory-based modelling aims at understanding and offers interpretability, on the other when modelling complex systems, simplifications and assumptions are inevitable, leading to inaccuracies or reduced utility (Hand, 2019;Rankin et al, 2020). In fact, relying on population-level statistics does not produce models capable of reproducing heterogeneous health outcomes (Chen et al, 2019a).…”

“…ehrGAN is developed for sequences of medical codes Che et al. It learns a transitional distribution, combining an Encoder-Decoder CNN (Rankin et al, 2020) with VCD . The ehrGAN generator is trained to decode a random vector mixed with the latent space representation of a real patient (See Panel 2).…”

Synthetic Observational Health Data with GANs: from slow adoption to a boom in medical research and ultimately digital twins?

“…They bound their representational power to correlations intelligible to the modeler, being prone to obscure inaccuracies. SD generated by these models hits a ceiling of utility (Rankin et al, 2020). In the ML field, generative models learn an approximation of the multi-modal distribution, from which we can draw synthetic samples (Goodfellow et al, 2014).…”

“…Having served its primary purpose, this wealth of detailed information can further benefit patient well-being by sustaining medical research and development. That is to say, improving the development life-cycle of Health Informatics (HI), the predictive accuracy of Machine Learning (ML) algorithms, or enabling discoveries in research on clinical decisions, triage decisions, inter-institution collaboration, and HI automation (Rudin et al, 2020;Rankin et al, 2020). Big health data is the underpinning of two prime objectives of precision medicine: individualization of patient interventions and inferring the workings of biological systems from high-level analysis (Capobianco, 2020).…”