Federated learning improves site performance in multicenter deep learning without data sharing

Sarma, Karthik V.; Harmon, Stephanie; Sanford, Thomas; Roth, Holger R.; Xu, Ziyue; Tetreault, Jesse; Xu, Daguang; Flores, Mona G.; Raman, Avi; Kulkarni, Rushikesh; Wood, Bradford J.; Choyke, Peter L.; Priester, Alan; Marks, Leonard S.; Raman, Steven S.; Enzmann, Dieter R.; Türkbey, Barış; Speier, William; Arnold, Corey

doi:10.1093/jamia/ocaa341

Cited by 144 publications

(83 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Studies that evaluate the performance of predictive models at multiple sites often involve objective tasks that are easy to replicate, such as image processing or analysis of laboratory tests 25 , 26 , 28 , 39 . In this respect, the current study addresses a non-trivial prediction task: the outcome is subjective and provider-dependent, the variables include non-objective findings such as orders given, and the overall admission rate is influenced by the local patient population.…”

Section: Discussionmentioning

confidence: 99%

“…Four main approaches are commonly used today to address the challenges of prediction across multiple sites: (1) Training and testing a model at each site separately in order to build site-specific models 11 , 21 – 24 ; (2) creating a centralized database of data from all sites in order to build a single uniform model 25 , 26 ; (3) applying a Federated Learning (FL) approach in which the model is trained collaboratively at each site, sharing model parameters but not medical data across sites 27 , 28 ; and (4) when customization is not feasible or when little variation exists between sites, a ready-made model that was trained at one or a few sites, can be used across all sites. This last approach is common practice with clinical risk scores such as the Centor risk score for Group-A Streptococcus pharyngitis or the PECARN algorithm for managing children with traumatic brain injury, to name a few 29 – 31 .…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Prediction across healthcare settings: a case study in predicting emergency department disposition

et al. 2021

View full text Add to dashboard Cite

Several approaches exist today for developing predictive models across multiple clinical sites, yet there is a lack of comparative data on their performance, especially within the context of EHR-based prediction models. We set out to provide a framework for prediction across healthcare settings. As a case study, we examined an ED disposition prediction model across three geographically and demographically diverse sites. We conducted a 1-year retrospective study, including all visits in which the outcome was either discharge-to-home or hospitalization. Four modeling approaches were compared: a ready-made model trained at one site and validated at other sites, a centralized uniform model incorporating data from all sites, multiple site-specific models, and a hybrid approach of a ready-made model re-calibrated using site-specific data. Predictions were performed using XGBoost. The study included 288,962 visits with an overall admission rate of 16.8% (7.9–26.9%). Some risk factors for admission were prominent across all sites (e.g., high-acuity triage emergency severity index score, high prior admissions rate), while others were prominent at only some sites (multiple lab tests ordered at the pediatric sites, early use of ECG at the adult site). The XGBoost model achieved its best performance using the uniform and site-specific approaches (AUC = 0.9–0.93), followed by the calibrated-model approach (AUC = 0.87–0.92), and the ready-made approach (AUC = 0.62–0.85). Our results show that site-specific customization is a key driver of predictive model performance.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Prediction across healthcare settings: a case study in predicting emergency department disposition

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Furthermore, distribution discrepancies in training data from these populations result in biases that are one of the major hindrances before generalizing ML approaches. Given the large volume and diverse data needed for model training, Federated Learning (FL) approaches may provide a novel opportunity for the future of ML applications (Rajendran et al, 2021;Sarma et al, 2021). FL is a collaborative ML training approach in which training data is not centralized and stays within organizational boundaries (Figure 1).…”

Section: Introductionmentioning

confidence: 99%

In the Pursuit of Privacy: The Promises and Predicaments of Federated Learning in Healthcare

Topaloglu

Morrell

Rajendran

et al. 2021

Front. Artif. Intell.

View full text Add to dashboard Cite

Artificial Intelligence and its subdomain, Machine Learning (ML), have shown the potential to make an unprecedented impact in healthcare. Federated Learning (FL) has been introduced to alleviate some of the limitations of ML, particularly the capability to train on larger datasets for improved performance, which is usually cumbersome for an inter-institutional collaboration due to existing patient protection laws and regulations. Moreover, FL may also play a crucial role in circumventing ML’s exigent bias problem by accessing underrepresented groups’ data spanning geographically distributed locations. In this paper, we have discussed three FL challenges, namely: privacy of the model exchange, ethical perspectives, and legal considerations. Lastly, we have proposed a model that could aide in assessing data contributions of a FL implementation. In light of the expediency and adaptability of using the Sørensen–Dice Coefficient over the more limited (e.g., horizontal FL) and computationally expensive Shapley Values, we sought to demonstrate a new paradigm that we hope, will become invaluable for sharing any profit and responsibilities that may accompany a FL endeavor.

show abstract

“…Examples include the 100 patient Prostate MR Image Segmentation (PROMISE12) challenge dataset [10] and the 60 patient NCI-ISBI (National Cancer Institute-International Symposium on Biomedical Imaging) Automated Segmentation of Prostate Structures (ASPS13) challenge dataset [11]. Other algorithms have been trained on institutionally developed local datasets that include between 100 and 650 studies [12][13][14][15]. Unfortunately, the development of research-quality prostate boundary annotations is challenging.…”

Section: Introductionmentioning

confidence: 99%

Harnessing clinical annotations to improve deep learning performance in prostate segmentation

et al. 2021

Self Cite

View full text Add to dashboard Cite

Purpose Developing large-scale datasets with research-quality annotations is challenging due to the high cost of refining clinically generated markup into high precision annotations. We evaluated the direct use of a large dataset with only clinically generated annotations in development of high-performance segmentation models for small research-quality challenge datasets. Materials and methods We used a large retrospective dataset from our institution comprised of 1,620 clinically generated segmentations, and two challenge datasets (PROMISE12: 50 patients, ProstateX-2: 99 patients). We trained a 3D U-Net convolutional neural network (CNN) segmentation model using our entire dataset, and used that model as a template to train models on the challenge datasets. We also trained versions of the template model using ablated proportions of our dataset, and evaluated the relative benefit of those templates for the final models. Finally, we trained a version of the template model using an out-of-domain brain cancer dataset, and evaluated the relevant benefit of that template for the final models. We used five-fold cross-validation (CV) for all training and evaluation across our entire dataset. Results Our model achieves state-of-the-art performance on our large dataset (mean overall Dice 0.916, average Hausdorff distance 0.135 across CV folds). Using this model as a pre-trained template for refining on two external datasets significantly enhanced performance (30% and 49% enhancement in Dice scores respectively). Mean overall Dice and mean average Hausdorff distance were 0.912 and 0.15 for the ProstateX-2 dataset, and 0.852 and 0.581 for the PROMISE12 dataset. Using even small quantities of data to train the template enhanced performance, with significant improvements using 5% or more of the data. Conclusion We trained a state-of-the-art model using unrefined clinical prostate annotations and found that its use as a template model significantly improved performance in other prostate segmentation tasks, even when trained with only 5% of the original dataset.

show abstract

Federated learning improves site performance in multicenter deep learning without data sharing

Cited by 144 publications

References 24 publications

Prediction across healthcare settings: a case study in predicting emergency department disposition

Prediction across healthcare settings: a case study in predicting emergency department disposition

In the Pursuit of Privacy: The Promises and Predicaments of Federated Learning in Healthcare

Harnessing clinical annotations to improve deep learning performance in prostate segmentation

Contact Info

Product

Resources

About