Demonstrating an approach for evaluating synthetic geospatial and temporal epidemiologic data utility: Results from analyzing &gt;1.8 million SARS-CoV-2 tests in the United States National COVID Cohort Collaborative (N3C)

Thomas, J. A. C.; Foraker, Randi E.; Zamstein, Noa; Payne, Philip R. O.; Wilcox, Adam B.

doi:10.1101/2021.07.06.21259051

Cited by 3 publications

(4 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Using synthetic data for model building could lower cost, reduce barriers to entry, ease external validation using datasets from multiple health‐care systems, and facilitate hypotheses generation of disease mechanisms. Various health systems are already using synthetic datasets for quality improvement and medical research 13,14,27,48 …”

Section: Discussionmentioning

confidence: 99%

“…Various health systems are already using synthetic datasets for quality improvement and medical research. 13,14,27,48 A study limitation is the predominance of male and White subjects in this cohort with potentially greater exposure to traumatic brain injury and post-traumatic stress disorder in combat veterans and applicability to a more diverse patient population or other health-care systems needs empirical testing. Our prior study showed that the performance of a machine learning model to predict AD onset using blood pressure trajectories trained using VA EHR data was similar when applied to University of Michigan EHR data even though the demographic compositions are different 12 .…”

Section: Conflict Of Interest Statementmentioning

confidence: 99%

“…to derive synthetic patient data from CDW. The software is designed to compute and preserve the original cohort's statistical properties and higher-order relationships and to create a synthetic analog cohort without any one-to-one correspondence between the original and synthetic patients 13,14,27 . To best mimic the real cohort, the ADAMS Platform provides the ability to select only those vari- CIs were derived from the limiting Gaussian processes and estimated asymptotic variances 25,28 .…”

Section: Synthetic Patient Data Generationmentioning

confidence: 99%

See 2 more Smart Citations

Predicting 5‐year dementia conversion in veterans with mild cognitive impairment

Irwin,

Tjandra,

et al. 2024

Alz & Dem Diag Ass & Dis Mo

View full text Add to dashboard Cite

INTRODUCTIONIdentifying mild cognitive impairment (MCI) patients at risk for dementia could facilitate early interventions. Using electronic health records (EHRs), we developed a model to predict MCI to all‐cause dementia (ACD) conversion at 5 years.METHODSCox proportional hazards model was used to identify predictors of ACD conversion from EHR data in veterans with MCI. Model performance (area under the receiver operating characteristic curve [AUC] and Brier score) was evaluated on a held‐out data subset.RESULTSOf 59,782 MCI patients, 15,420 (25.8%) converted to ACD. The model had good discriminative performance (AUC 0.73 [95% confidence interval (CI) 0.72–0.74]), and calibration (Brier score 0.18 [95% CI 0.17–0.18]). Age, stroke, cerebrovascular disease, myocardial infarction, hypertension, and diabetes were risk factors, while body mass index, alcohol abuse, and sleep apnea were protective factors.DISCUSSIONEHR‐based prediction model had good performance in identifying 5‐year MCI to ACD conversion and has potential to assist triaging of at‐risk patients.Highlights Of 59,782 veterans with mild cognitive impairment (MCI), 15,420 (25.8%) converted to all‐cause dementia within 5 years. Electronic health record prediction models demonstrated good performance (area under the receiver operating characteristic curve 0.73; Brier 0.18). Age and vascular‐related morbidities were predictors of dementia conversion. Synthetic data was comparable to real data in modeling MCI to dementia conversion. Key Points An electronic health record–based model using demographic and co‐morbidity data had good performance in identifying veterans who convert from mild cognitive impairment (MCI) to all‐cause dementia (ACD) within 5 years. Increased age, stroke, cerebrovascular disease, myocardial infarction, hypertension, and diabetes were risk factors for 5‐year conversion from MCI to ACD. High body mass index, alcohol abuse, and sleep apnea were protective factors for 5‐year conversion from MCI to ACD. Models using synthetic data, analogs of real patient data that retain the distribution, density, and covariance between variables of real patient data but are not attributable to any specific patient, performed just as well as models using real patient data. This could have significant implications in facilitating widely distributed computing of health‐care data with minimized patient privacy concern that could accelerate scientific discoveries.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Conflict Of Interest Statementmentioning

confidence: 99%

Section: Synthetic Patient Data Generationmentioning

confidence: 99%

See 1 more Smart Citation

Predicting 5‐year dementia conversion in veterans with mild cognitive impairment

Irwin,

Tjandra,

et al. 2024

Alz & Dem Diag Ass & Dis Mo

View full text Add to dashboard Cite

show abstract

“…The VHA data is enriched for individuals whose privacy is a matter of national security concern. Additionally, synthetic COVID-19 patients were demonstrated to be an effective surrogate for a variety of public health and clinical tasks ( 23 , 24 ). Therefore, synthetic data may play a critical role in modeling COVID-19 and developing AI applications in general within the VHA.…”

Section: Introductionmentioning

confidence: 99%

Synthetic Health Data Can Augment Community Research Efforts to Better Inform the Public During Emerging Pandemics

Prasanna,

Jing,

Plopper

et al. 2023

Preprint

View full text Add to dashboard Cite

The COVID-19 pandemic had disproportionate effects on the Veteran population due to the increased prevalence of medical and environmental risk factors. Synthetic electronic health record (EHR) data can help meet the acute need for Veteran population-specific predictive modeling efforts by avoiding the strict barriers to access, currently present within Veteran Health Administration (VHA) datasets. The U.S. Food and Drug Administration (FDA) and the VHA launched the precisionFDA COVID-19 Risk Factor Modeling Challenge to develop COVID-19 diagnostic and prognostic models; identify Veteran population-specific risk factors; and test the usefulness of synthetic data as a substitute for real data. The use of synthetic data boosted challenge participation by providing a dataset that was accessible to all competitors. Models trained on synthetic data showed similar but systematically inflated model performance metrics to those trained on real data. The important risk factors identified in the synthetic data largely overlapped with those identified from the real data, and both sets of risk factors were validated in the literature. Tradeoffs exist between synthetic data generation approaches based on whether a real EHR dataset is required as input. Synthetic data generated directly from real EHR input will more closely align with the characteristics of the relevant cohort. This work shows that synthetic EHR data will have practical value to the Veterans’ health research community for the foreseeable future.

show abstract

A Scoping Review of Privacy and Utility Metrics in Medical Synthetic Data

Kaabachi,

Despraz,

Meurers

et al. 2023

Preprint

View full text Add to dashboard Cite

IntroductionSharing and re-using health-related data beyond the scope of its initial collection is essential for accelerating research, developing robust and trustworthy machine learning algorithms methods that can be translated into clinical settings. The sharing of synthetic data, artificially generated to resemble real patient data, is increasingly recognized as a promising means to enable such a re-use while addressing the privacy concerns related to personal medical data. Nonetheless, no consensus exists yet on a standard approach for systematically and quantitatively evaluating the actual privacy gain and residual utility of synthetic data, de-facto hindering its adoption.ObjectiveIn this work, we present and systematize current knowledge on the field of synthetic health-related data evaluation both in terms of privacy and utility. We provide insights and critical analysis into the current state of the art and propose concrete directions and steps forward for the research community.MethodsWe assess and contextualize existing knowledge in the field through a scoping review and the creation of a common ontology that encompasses all the methods and metrics used to assess synthetic data. We follow the PRISMA-ScR methodology in order to perform data collection and knowledge synthesis.ResultsWe include 92 studies in the scoping review. We analyze and classify them according to the proposed ontology. We found 48 different methods to evaluate the residual statistical utility of synthetic data and 9 methods that are used to evaluate the residual privacy risks. Moreover, we observe that there is currently no consensus among researchers regarding neither individual metrics nor family of metrics for evaluating the privacy and utility of synthetic data. Our findings on the privacy of synthetic data show that there is an alarming tendency to trust the safety of synthetic data without properly evaluating it.ConclusionAlthough the use of synthetic data in healthcare promises to offer an easy and hassle-free alternative to real data, the lack of consensus in terms of evaluation hinders the adoption of this new technology. We believe that, by raising awareness and providing a comprehensive taxonomy on evaluation methods that takes into account the current state of literature, our work can foster the development and adoption of uniform approaches and consequently facilitate the use of synthetic data in the medical domain.

show abstract

Demonstrating an approach for evaluating synthetic geospatial and temporal epidemiologic data utility: Results from analyzing >1.8 million SARS-CoV-2 tests in the United States National COVID Cohort Collaborative (N3C)

Cited by 3 publications

References 34 publications

Predicting 5‐year dementia conversion in veterans with mild cognitive impairment

Predicting 5‐year dementia conversion in veterans with mild cognitive impairment

Synthetic Health Data Can Augment Community Research Efforts to Better Inform the Public During Emerging Pandemics

A Scoping Review of Privacy and Utility Metrics in Medical Synthetic Data

Contact Info

Product

Resources

About