Pretreatment prediction for IVF outcomes: generalized applicable model or centre-specific model?

Abstract

STUDY QUESTION What was the performance of different pretreatment prediction models for IVF, which were developed based on UK/US population (McLernon 2016 model, Luke model, Dhillon model, and McLernon 2022 model), in wider populations? SUMMARY ANSWER For a patient in China, the published pretreatment prediction models based on the UK/US population provide similar discriminatory power with reasonable AUCs and underestimated predictions. WHAT IS KNOWN ALREADY Several pretreatment prediction models for IVF allow patients and clinicians to estimate the cumulative probability of live birth in a cycle before the treatment, but they are mostly based on the population of Europe or the USA, and their performance and applicability in the countries and regions beyond these regions are largely unknown. STUDY DESIGN, SIZE, DURATION A total of 26 382 Chinese patients underwent oocyte pick-up cycles between January 2013 and December 2020. PARTICIPANTS/MATERIALS, SETTING, METHODS UK/US model performance was externally validated according to the coefficients and intercepts they provided. Centre-specific models were established with XGboost, Lasso, and generalized linear model algorithms. Discriminatory power and calibration of the models were compared as the forms of the AUC of the Receiver Operator Characteristic and calibration curves. MAIN RESULTS AND THE ROLE OF CHANCE The AUCs for McLernon 2016 model, Luke model, Dhillon model, and McLernon 2022 model were 0.69 (95% CI 0.68–0.69), 0.67 (95% CI 0.67–0.68), 0.69 (95% CI 0.68–0.69), and 0.67 (95% CI 0.67–0.68), respectively. The centre-specific yielded an AUC of 0.71 (95% CI 0.71–0.72) with key predictors including age, duration of infertility, and endocrine parameters. All external models suggested underestimation. Among the external models, the rescaled McLernon 2022 model demonstrated the best calibration (Slope 1.12, intercept 0.06). LIMITATIONS, REASONS FOR CAUTION The study is limited by its single-centre design and may not be representative elsewhere. Only per-complete cycle validation was carried out to provide a similar framework to compare different models in the sample population. Newer predictors, such as AMH, were not used. WIDER IMPLICATIONS OF THE FINDINGS Existing pretreatment prediction models for IVF may be used to provide useful discriminatory power in populations different from those on which they were developed. However, models based on newer more relevant datasets may provide better calibrations. STUDY FUNDING/COMPETING INTEREST(S) This work was supported by the National Natural Science Foundation of China [grant number 22176159], the Xiamen Medical Advantage Subspecialty Construction Project [grant number 2018296], and the Special Fund for Clinical and Scientific Research of Chinese Medical Association [grant number 18010360765]. TRIAL REGISTRATION NUMBER N/A.

Improving IVF Utilization with Patient-Centric Artificial Intelligence-Machine Learning (AI/ML): A Retrospective Multicenter Experience

“…Though national registries have provided a bird’s eye view of IVF outcomes to inform policy and maternal and newborn safety guidelines, the Univfy report enables providers to communicate the IVF live birth probability based on each patient’s own health data with a prediction model that has been validated for data from their specific fertility center. The practical clinical benefits of using the machine learning, center-specific approach specific to model performance are discussed in other reports and are beyond the scope of this article [ 19 , 20 , 21 , 22 , 23 , 24 , 25 , 29 , 30 , 31 , 33 ].…”

“…For example, online calculators have been developed using the US Society for Reproductive Technology (SART) or UK Human Fertilization and Embryology Authority (HFEA) national database without center-specific validation [26][27][28]. Alternatively, other researchers- Qiu et al, 2019;Liu et al, 2013;Cai et al, 2024-have also reported using machine learning to train and validate IVF pre-treatment models with excellent, center-specific validation results [29][30][31]. Finally, some IVF prediction models such as that reported by Wen et al, 2022 contributed to our insights but would not practically be usable for patient counseling in the pre-treatment context, since they required data that are not available prior to starting IVF (e.g., number of oocytes, number of blastocysts, etc.)…”

Predicting IVF live birth probabilities using machine learning, center-specific models: validation results and potential benefits over national registry-based models

“…Overall, the MLCS approach is expected to provide more locally relevant prognostic information as it is unaffected by inter-center variations in patients' attributes and clinical or embryology laboratory protocols. (1,(4)(5)(9)(10)(11)(12) Further, conventional ML methods have remained comparable or even superior to deep learning methods when applied to train structured healthcare data. (13) Nonetheless, there is a perception that multicenter, registry-based IVF prognostics models as exemplified by the "McLernon models" --US Society for Assisted Reproductive Technology (SART) pretreatment model (aka SART calculator or SART model) and the "UK McLernon 2022 model" --are All rights reserved.…”

“…(4)(5)(14)(15)(16) Using a single center's dataset comprising ~26K+ IVF cycles, Cai et al reported improved and more locally relevant model performance using the MLCS approach, refuting the recommendation by McLernon et al to develop center-specific models by recalibrating from the SART or UK McLernon models. (9) Many US providers have asked us to show the differential prognostic information provided by the SART calculator and an MLCS model and whether MLCS is applicable to small-tomidsize US fertility centers with much lower IVF volumes compared to the report by Cai et al However, a head-to-head comparison between the MLCS and McLernon pretreatment models has not been performed for centers reporting to the US or UK registries. Addressing these questions will help us to develop best practices for IVF prognostic counseling, which is critical for advancing and expanding fertility care in the US and globally.…”

“…Overall, the MLCS approach is expected to provide more locally relevant prognostic information as it is unaffected by inter-center variations in patients’ attributes and clinical or embryology laboratory protocols. (1, 4–5, 9–12) Further, conventional ML methods have remained comparable or even superior to deep learning methods when applied to train structured healthcare data. (13)…”