Evaluating the Impact of Health Care Data Completeness for Deep Generative Models

Abstract

Background: Deep generative models (DGMs) present a promising avenue for generating realistic, synthetic data to augment existing healthcare datasets. However, exactly how the completeness of the original dataset affects the quality of the generated synthetic data is unclear. Objectives: In this paper, we investigate the effect of data completeness on samples generated by the most common DGM paradigms. Methods: We create both cross-sectional and panel datasets with varying missingness and subset rates and train generative adversarial networks (GANs), variational autoencoders (VAEs) and autoregressive models (Transformers) on these datasets. We then compare the distributions of generated data with original training data to measure similarity. Results: We find that increased incompleteness is directly correlated with increased dissimilarity between original and generated samples produced through DGMs. Conclusions: Care must be taken when using DGMs to generate synthetic data as data completeness issues can affect the quality of generated data in both panel and cross-sectional datasets.

High-Quality Data for Health Care and Health Research

“…11 Tahar et al decline the use of the term “consistency” for their study and refer to plausibility instead, broadly defined as “deviations from expected values.” 14 Smith et al talk about “consistency and similarity of synthetic samples” without precisely mentioning their interpretation of the dimension. 16 Beyond the different interpretations of terms that are directly related to the topic data quality, one might imagine similar deviations regarding basic concepts such as “data set” or “metadata.” Tahar et al looked at those terms and propose respective definitions. 14 Nevertheless, the manifold perspectives and interpretations of domain-specific and basic terms will impede the distribution of the concepts, frameworks, packages, and tools presented by the papers to a wider application.…”

“…14 The two remaining papers picked up the approach of synthetic data collections. 15,16 Concepts Related to Data Quality Mashoufi et al present a scoping review about the main concepts related to data quality and data quality assessment methodologies. 9 In their search for these concepts, they focused on health care using the search term "medical record" and considered both paper-based and computerized records.…”

“…Using synthetic data, Smith et al analyzed the effect of different degrees of data and case completeness with regard to machine learning services. 16 Case completeness is particularly critical in that domain due to a potential bias that might occur if minority populations are not present. The modified synthetic data were utilized by three different deep generative frameworks.…”

“…Tahar et al decline the use of the term "consistency" for their study and refer to plausibility instead, broadly defined as "deviations from expected values. "14 Smith et al talk about "consistency and similarity of synthetic samples" without precisely mentioning their interpretation of the dimension 16. Beyond the different interpretations of terms that are directly related to the topic data quality, one might imagine similar deviations regarding basic concepts such as "data set" or "metadata."…”