Guidelines, Consensus Statements, and Standards for the Use of Artificial Intelligence in Medicine: Systematic Review

Wang, Ying; Li, Nian; Chen, Lingmin; Wu, Miaomiao; Meng, Sha; Dai, Zelei; Zhang, Yonggang; Clarke, Mike

doi:10.2196/46089

Cited by 12 publications

(1 citation statement)

References 52 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the era of generative conversational artificial intelligence (AI) [ 9 , 10 ], characterized by disruptive technological transformation, the importance of sleep health promotion and education becomes even more relevant [ 11 ]: conversational AI-based platforms and agents, such as chatbots, can provide instant responses to sleep-related queries, making information readily available at any time. This real-time accessibility can help individuals seeking answers about sleep health, who can receive personalized advice and recommendations based on an individual’s specific sleep patterns and concerns.…”

Section: Introductionmentioning

confidence: 99%

Assessing the Accuracy of Generative Conversational Artificial Intelligence in Debunking Sleep Health Myths: Mixed Methods Comparative Study With Expert Analysis

Bragazzi,

Garbarino

2024

JMIR Form Res

View full text Add to dashboard Cite

Background Adequate sleep is essential for maintaining individual and public health, positively affecting cognition and well-being, and reducing chronic disease risks. It plays a significant role in driving the economy, public safety, and managing health care costs. Digital tools, including websites, sleep trackers, and apps, are key in promoting sleep health education. Conversational artificial intelligence (AI) such as ChatGPT (OpenAI, Microsoft Corp) offers accessible, personalized advice on sleep health but raises concerns about potential misinformation. This underscores the importance of ensuring that AI-driven sleep health information is accurate, given its significant impact on individual and public health, and the spread of sleep-related myths. Objective This study aims to examine ChatGPT’s capability to debunk sleep-related disbeliefs. Methods A mixed methods design was leveraged. ChatGPT categorized 20 sleep-related myths identified by 10 sleep experts and rated them in terms of falseness and public health significance, on a 5-point Likert scale. Sensitivity, positive predictive value, and interrater agreement were also calculated. A qualitative comparative analysis was also conducted. Results ChatGPT labeled a significant portion (n=17, 85%) of the statements as “false” (n=9, 45%) or “generally false” (n=8, 40%), with varying accuracy across different domains. For instance, it correctly identified most myths about “sleep timing,” “sleep duration,” and “behaviors during sleep,” while it had varying degrees of success with other categories such as “pre-sleep behaviors” and “brain function and sleep.” ChatGPT’s assessment of the degree of falseness and public health significance, on the 5-point Likert scale, revealed an average score of 3.45 (SD 0.87) and 3.15 (SD 0.99), respectively, indicating a good level of accuracy in identifying the falseness of statements and a good understanding of their impact on public health. The AI-based tool showed a sensitivity of 85% and a positive predictive value of 100%. Overall, this indicates that when ChatGPT labels a statement as false, it is highly reliable, but it may miss identifying some false statements. When comparing with expert ratings, high intraclass correlation coefficients (ICCs) between ChatGPT’s appraisals and expert opinions could be found, suggesting that the AI’s ratings were generally aligned with expert views on falseness (ICC=.83, P<.001) and public health significance (ICC=.79, P=.001) of sleep-related myths. Qualitatively, both ChatGPT and sleep experts refuted sleep-related misconceptions. However, ChatGPT adopted a more accessible style and provided a more generalized view, focusing on broad concepts, while experts sometimes used technical jargon, providing evidence-based explanations. Conclusions ChatGPT-4 can accurately address sleep-related queries and debunk sleep-related myths, with a performance comparable to sleep experts, even if, given its limitations, the AI cannot completely replace expert opinions, especially in nuanced and complex fields such as sleep health, but can be a valuable complement in the dissemination of updated information and promotion of healthy behaviors.

show abstract

Section: Introductionmentioning

confidence: 99%

Assessing the Accuracy of Generative Conversational Artificial Intelligence in Debunking Sleep Health Myths: Mixed Methods Comparative Study With Expert Analysis

Bragazzi,

Garbarino

2024

JMIR Form Res

View full text Add to dashboard Cite

show abstract

Données synthétiques en médecine : génération, évaluation et limites

Benani,

Vibert,

Demuth

2024

Med Sci (Paris)

View full text Add to dashboard Cite

show abstract

CALIFRAME: a proposed method of calibrating reporting guidelines with FAIR principles to foster reproducibility of AI research in medicine

Shiferaw,

Balaur,

Welter

et al. 2024

JAMIA Open

View full text Add to dashboard Cite

Background Procedural and reporting guidelines are crucial in framing scientific practices and communications among researchers and the broader community. These guidelines aim to ensure transparency, reproducibility, and reliability in scientific research. Despite several methodological frameworks proposed by various initiatives to foster reproducibility, challenges such as data leakage and reproducibility remain prevalent. Recent studies have highlighted the transformative potential of incorporating the FAIR (Findable, Accessible, Interoperable, and Reusable) principles into workflows, particularly in contexts like software and machine learning model development, to promote open science. Objective This study aims to introduce a comprehensive framework, designed to calibrate existing reporting guidelines against the FAIR principles. The goal is to enhance reproducibility and promote open science by integrating these principles into the scientific reporting process. Methods We employed the “Best fit” framework synthesis approach which involves systematically reviewing and synthesizing existing frameworks and guidelines to identify best practices and gaps. We then proposed a series of defined workflows to align reporting guidelines with FAIR principles. A use case was developed to demonstrate the practical application of the framework. Results The integration of FAIR principles with established reporting guidelines through the framework effectively bridges the gap between FAIR metrics and traditional reporting standards. The framework provides a structured approach to enhance the findability, accessibility, interoperability, and reusability of scientific data and outputs. The use case demonstrated the practical benefits of the framework, showing improved data management and reporting practices. Discussion The framework addresses critical challenges in scientific research, such as data leakage and reproducibility issues. By embedding FAIR principles into reporting guidelines, the framework ensures that scientific outputs are more transparent, reliable, and reusable. This integration not only benefits researchers by improving data management practices but also enhances the overall scientific process by promoting open science and collaboration. Conclusion The proposed framework successfully combines FAIR principles with reporting guidelines, offering a robust solution to enhance reproducibility and open science. This framework can be applied across various contexts, including software and machine learning model development stages, to foster a more transparent and collaborative scientific environment.

show abstract

Guidelines, Consensus Statements, and Standards for the Use of Artificial Intelligence in Medicine: Systematic Review

Cited by 12 publications

References 52 publications

Assessing the Accuracy of Generative Conversational Artificial Intelligence in Debunking Sleep Health Myths: Mixed Methods Comparative Study With Expert Analysis

Assessing the Accuracy of Generative Conversational Artificial Intelligence in Debunking Sleep Health Myths: Mixed Methods Comparative Study With Expert Analysis

Données synthétiques en médecine : génération, évaluation et limites

CALIFRAME: a proposed method of calibrating reporting guidelines with FAIR principles to foster reproducibility of AI research in medicine

Contact Info

Product

Resources

About