Reproducibility of prediction models in health services research

Belbasis, Lazaros; Panagiotou, Orestis A.

doi:10.1186/s13104-022-06082-4

Cited by 6 publications

(3 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Reproducibility, described as "the ability of an independent research team to produce the same results using the AI method based on the documentation made by the original research team" [15], requires an exact representation of all relevant aspects of the study development and realization. This includes the complete information of the used software and source code, the original data as well as the correct documentation of crucial details and precise instructions for the implementation [39][40][41]. The reproducibility in AI builds trust in the developed models and results [14,40].…”

Section: Contributions To Better Reproducibility In Ai In Medicine An...mentioning

confidence: 99%

Guidelines and Standard Frameworks for Artificial Intelligence in Medicine: A Systematic Review

Shiferaw,

Roloff,

Balaur

et al. 2024

Preprint

View full text Add to dashboard Cite

A growing volume of evidence marks the potential of Artificial Intelligence (AI) in medicine, in improving diagnostic accuracy, clinical decision support, risk/event prediction, drug discovery, and patient management. However, the continuous integration of AI into clinical settings requires the development of up-to-date and robust guidelines and standard frameworks that consider the evolving challenges of AI implementation in medicine. This review evaluates these guidelines quality and summarizes ethical frameworks, best practices, and recommendations. The Appraisal of Guidelines, Research, and Evaluation (AGREE II) tool was used to assess the quality of guidelines based on six domains: scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence. The protocol of this review including the eligibility criteria, the search strategy data extraction sheet and methods, was published prior to the actual review with International Registered Report Identifier (IRRID) of (DERR1-10.2196/47105). The initial search resulted in 4,975 studies from two databases and five studies from manual search. Nine articles were selected for data extraction based on the eligibility criteria. We found that while guidelines generally excel in scope, purpose, and editorial independence, there is significant variability in applicability and the rigour of guideline development. Well-established initiatives such as DECIDE-AI, SPIRIT-AI, and CONSORT-AI have shown high quality, particularly in terms of stakeholder involvement. However, applicability remains a prominent challenge among the guidelines. We conclude that the reproducibility, ethical and environmental aspects of AI in medicine still need attention from both medical and AI communities. This review emphasizes the crucial need for high-quality guidelines and opens a new avenue in evaluating guidelines themselves. Our work highlights the need for working toward the development of integrated and comprehensive reporting guidelines that adhere to the principles of Findability, Accessibility, Interoperability and Reusability (FAIR). This alignment is essential for fostering a cultural shift towards transparency and open science, which are pivotal milestone for sustainable digital health research.

show abstract

Section: Contributions To Better Reproducibility In Ai In Medicine An...mentioning

confidence: 99%

Guidelines and Standard Frameworks for Artificial Intelligence in Medicine: A Systematic Review

Shiferaw,

Roloff,

Balaur

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…These models are mostly statistical and often result from a process that seeks those risk factors or symptoms that correlate most with the health issue being modelled. In general, such modelling usually suffers from the following limitations: A focus on predicting presence of a condition of interest without considering knowledge of absence of that condition Lack of access to sufficient high quality data, resulting in low quality predictions in the presence of uncertain or missing data. Use of internal statistical validation that evaluates the model’s prediction but is unable to evaluate the structure or reasoning process. Limited case vignettes with exemplar predictions that would help readers relate to the model and compare it to other predictive and diagnostic approaches. The lower methodological quality and poor reproducibility of many models [1, 2], and authors who simply present the same model in many articles with only minor differences in the cohort [3], have led some to claim that predictive models amount to little more than junk science [4, 5]. While the focus of this study is on the problem of predicting pregnancy outcomes, the proposed approach is sufficiently general that it can be used for creating and validating holistic whole-of-condition predictive models for a wide range of medical conditions.…”

Section: Introductionmentioning

confidence: 99%

“…The lower methodological quality and poor reproducibility of many models [1,2], and authors who simply present the same model in many articles with only minor differences in the cohort [3], have led some to claim that predictive models amount to little more than junk science [4,5]. While the focus of this study is on the problem of predicting pregnancy outcomes, the proposed approach is sufficiently general that it can be used for creating and validating holistic whole-ofcondition predictive models for a wide range of medical conditions.…”

mentioning

confidence: 99%

Approach and Method for Bayesian Network Modelling: A Case Study in Pregnancy Outcomes for England and Wales

McLachlan,

Daley,

Saidi

et al. 2024

Preprint

View full text Add to dashboard Cite

Efforts to fully exploit the rich potential of Bayesian Networks (BNs) have hitherto not seen a practical approach for development of domain-specific models using large-scale public statistics which have the potential to reduce the time required to develop probability tables and train the model. As a result, the duration of projects seeking to develop health BNs tend to be measured in years due to their reliance on obtaining ethics approval and collecting, normalising, and discretising collections of patient EHRs. This work addresses this challenge by investigating a new approach to developing health BNs that combines expert elicitation with knowledge from literature and national health statistics. The approach presented here is evaluated through the development of a BN for pregnancy complications and outcomes using national health statistics for all births in England and Wales during 2021. The result is a BN that when validated using vignettes against other common types of predictive models including multivariable logistic regression and nomograms produces comparable predictions. The BN using our approach and large-scale public statistics was also developed in a project with a duration measured in months rather than years. The unique contributions of this paper are a new efficient approach to BN development and a working BN capable of reasoning over a broad range of pregnancy-related conditions and outcomes.

show abstract

Prediction models using artificial intelligence and longitudinal data from electronic health records: a systematic methodological review

Carrasco-Ribelles,

Llanes-Jurado,

Gallego-Moll

et al. 2023

Journal of the American Medical Informatics Association

View full text Add to dashboard Cite

Objective To describe and appraise the use of artificial intelligence (AI) techniques that can cope with longitudinal data from electronic health records (EHRs) to predict health-related outcomes. Methods This review included studies in any language that: EHR was at least one of the data sources, collected longitudinal data, used an AI technique capable of handling longitudinal data, and predicted any health-related outcomes. We searched MEDLINE, Scopus, Web of Science, and IEEE Xplorer from inception to January 3, 2022. Information on the dataset, prediction task, data preprocessing, feature selection, method, validation, performance, and implementation was extracted and summarized using descriptive statistics. Risk of bias and completeness of reporting were assessed using a short form of PROBAST and TRIPOD, respectively. Results Eighty-one studies were included. Follow-up time and number of registers per patient varied greatly, and most predicted disease development or next event based on diagnoses and drug treatments. Architectures generally were based on Recurrent Neural Networks-like layers, though in recent years combining different layers or transformers has become more popular. About half of the included studies performed hyperparameter tuning and used attention mechanisms. Most performed a single train-test partition and could not correctly assess the variability of the model’s performance. Reporting quality was poor, and a third of the studies were at high risk of bias. Conclusions AI models are increasingly using longitudinal data. However, the heterogeneity in reporting methodology and results, and the lack of public EHR datasets and code sharing, complicate the possibility of replication. Registration PROSPERO database (CRD42022331388).

show abstract

Reproducibility of prediction models in health services research

Cited by 6 publications

References 49 publications

Guidelines and Standard Frameworks for Artificial Intelligence in Medicine: A Systematic Review

Guidelines and Standard Frameworks for Artificial Intelligence in Medicine: A Systematic Review

Approach and Method for Bayesian Network Modelling: A Case Study in Pregnancy Outcomes for England and Wales

Prediction models using artificial intelligence and longitudinal data from electronic health records: a systematic methodological review

Contact Info

Product

Resources

About