Evaluating ChatGPT responses on obstructive sleep apnea for patient education

Campbell, Daniel; Estephan, Leonard; Mastrolonardo, Eric; Amin, Dev; Huntley, Colin; Boon, Maurits

doi:10.5664/jcsm.10728

Cited by 36 publications

(9 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…With open-ended questions, ChatGPT can give as much information as possible by retrieving from its trained database. Our findings echo a prior study by Campbell et al (2023) that found ChatGPT provided appropriate answers to most questions about obstructive sleep apnea regardless of prompting. Since many first-time users may not be aware that ChatGPT can be fine-tuned with different styles of initial prompts, it is reassuring to know that patients who query ChatGPT are likely to receive a clinically-appropriate response.…”

Section: Discussionsupporting

confidence: 89%

Comparing ChatGPT and clinical nurses’ performances on tracheostomy care: A cross-sectional study

Wang,

Mu,

Chen

et al. 2024

International Journal of Nursing Studies Advances

View full text Add to dashboard Cite

Section: Discussionsupporting

confidence: 89%

Comparing ChatGPT and clinical nurses’ performances on tracheostomy care: A cross-sectional study

Wang,

Mu,

Chen

et al. 2024

International Journal of Nursing Studies Advances

View full text Add to dashboard Cite

“…The responses that would signify a large deficiency -incorrect or incomplete (greater than 50% missing) information -were only present in 5.5% of responses. This result was on par with many other studies that examined AI chatbot responses, generally ranging from 1% to 5% incorrect responses [13][14][15][16]. ChatGPT gave more partially correct answers than Gemini, while Gemini exhibited a non-significant trend to provide more correct responses than did ChatGPT.…”

Section: Discussionsupporting

confidence: 68%

Unlocking Health Literacy: The Ultimate Guide to Hypertension Education From ChatGPT Versus Google Gemini

Lee,

Campbell,

Patel

et al. 2024

Cureus

View full text Add to dashboard Cite

Background Google Gemini (Google, Mountain View, CA) represents the latest advances in the realm of artificial intelligence (AI) and has garnered attention due to its capabilities similar to the increasingly popular ChatGPT (OpenAI, San Francisco, CA). Accurate dissemination of information on common conditions such as hypertension is critical for patient comprehension and management. Despite the ubiquity of AI, comparisons between ChatGPT and Gemini remain unexplored. Methods ChatGPT and Gemini were asked 52 questions derived from the American College of Cardiology’s (ACC) frequently asked questions on hypertension, following a specified prompt. Prompts included: no prompting (Form 1), patient-friendly prompting (Form 2), physician-level prompting (Form 3), and prompting for statistics/references (Form 4). Responses were scored as incorrect, partially correct, or correct. Flesch-Kincaid (FK) grade level and word count were recorded. Results Across all forms, scoring frequencies were as follows: 23 (5.5%) incorrect, 162 (38.9%) partially correct, and 231 (55.5%) correct. ChatGPT showed higher rates of partially correct answers than Gemini (p = 0.0346). Physician-level prompts resulted in a higher word count across both platforms (p < 0.001). ChatGPT showed a higher FK grade level (p = 0.033) in physician-friendly prompting. Gemini exhibited a significantly higher mean word count (p < 0.001); however, ChatGPT had a higher FK grade level across all forms (p > 0.001). Conclusion To our knowledge, this study is the first to compare cardiology-related responses from ChatGPT and Gemini, two of the most popular AI chatbots. The grade level for most responses was collegiate level, which was above average for the National Institutes of Health (NIH) recommendations, but on par with most online medical information. Both chatbots responded with a high degree of accuracy, with inaccuracies being rare. Therefore, it is reasonable that cardiologists suggest either chatbot as a source of supplementary education.

show abstract

“…There are only a few studies that have assessed sleep-related knowledge of conversational AI-based chatbots, such as ChatGPT-4, which was found very recently to successfully pass the sleep medicine certification board examinations [ 13 ] and be conversant in sleep disorders, such as obstructive sleep apnea syndrome [ 14 - 16 ].…”

Section: Introductionmentioning

confidence: 99%

Assessing the Accuracy of Generative Conversational Artificial Intelligence in Debunking Sleep Health Myths: Mixed Methods Comparative Study With Expert Analysis

Bragazzi,

Garbarino

2024

JMIR Form Res

View full text Add to dashboard Cite

Background Adequate sleep is essential for maintaining individual and public health, positively affecting cognition and well-being, and reducing chronic disease risks. It plays a significant role in driving the economy, public safety, and managing health care costs. Digital tools, including websites, sleep trackers, and apps, are key in promoting sleep health education. Conversational artificial intelligence (AI) such as ChatGPT (OpenAI, Microsoft Corp) offers accessible, personalized advice on sleep health but raises concerns about potential misinformation. This underscores the importance of ensuring that AI-driven sleep health information is accurate, given its significant impact on individual and public health, and the spread of sleep-related myths. Objective This study aims to examine ChatGPT’s capability to debunk sleep-related disbeliefs. Methods A mixed methods design was leveraged. ChatGPT categorized 20 sleep-related myths identified by 10 sleep experts and rated them in terms of falseness and public health significance, on a 5-point Likert scale. Sensitivity, positive predictive value, and interrater agreement were also calculated. A qualitative comparative analysis was also conducted. Results ChatGPT labeled a significant portion (n=17, 85%) of the statements as “false” (n=9, 45%) or “generally false” (n=8, 40%), with varying accuracy across different domains. For instance, it correctly identified most myths about “sleep timing,” “sleep duration,” and “behaviors during sleep,” while it had varying degrees of success with other categories such as “pre-sleep behaviors” and “brain function and sleep.” ChatGPT’s assessment of the degree of falseness and public health significance, on the 5-point Likert scale, revealed an average score of 3.45 (SD 0.87) and 3.15 (SD 0.99), respectively, indicating a good level of accuracy in identifying the falseness of statements and a good understanding of their impact on public health. The AI-based tool showed a sensitivity of 85% and a positive predictive value of 100%. Overall, this indicates that when ChatGPT labels a statement as false, it is highly reliable, but it may miss identifying some false statements. When comparing with expert ratings, high intraclass correlation coefficients (ICCs) between ChatGPT’s appraisals and expert opinions could be found, suggesting that the AI’s ratings were generally aligned with expert views on falseness (ICC=.83, P<.001) and public health significance (ICC=.79, P=.001) of sleep-related myths. Qualitatively, both ChatGPT and sleep experts refuted sleep-related misconceptions. However, ChatGPT adopted a more accessible style and provided a more generalized view, focusing on broad concepts, while experts sometimes used technical jargon, providing evidence-based explanations. Conclusions ChatGPT-4 can accurately address sleep-related queries and debunk sleep-related myths, with a performance comparable to sleep experts, even if, given its limitations, the AI cannot completely replace expert opinions, especially in nuanced and complex fields such as sleep health, but can be a valuable complement in the dissemination of updated information and promotion of healthy behaviors.

show abstract

Evaluating ChatGPT responses on obstructive sleep apnea for patient education

Cited by 36 publications

References 10 publications

Comparing ChatGPT and clinical nurses’ performances on tracheostomy care: A cross-sectional study

Comparing ChatGPT and clinical nurses’ performances on tracheostomy care: A cross-sectional study

Unlocking Health Literacy: The Ultimate Guide to Hypertension Education From ChatGPT Versus Google Gemini

Assessing the Accuracy of Generative Conversational Artificial Intelligence in Debunking Sleep Health Myths: Mixed Methods Comparative Study With Expert Analysis

Contact Info

Product

Resources

About