How AI Responds to Common Lung Cancer Questions: ChatGPT versus                     Google Bard

Rahsepar, Amir Ali; Tavakoli, Neda; Kim, Hyun J; Hassani, Cameron; Abtin, Fereidoun; Bedayat, Arash

doi:10.1148/radiol.230922

Cited by 162 publications

(46 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similar results were also reported in another recently published study assessing these three LLMs in a different cancer-related topic. Specifically, Rahsepar et al reported the results of a study investigating the ability of ChatGPT-3.5, ChatGPT-4 and Google Bard in answering questions related to lung cancer screening and prevention (21). As in our study, ChatGPT achieved a superior performance to Google Bard.…”

Section: Discussionmentioning

confidence: 99%

Comparison of Large Language Models in Answering Immuno-Oncology Questions: A Cross-Sectional Study

Iannantuono,

Bracken-Clarke,

Karzai

et al. 2023

Preprint

View full text Add to dashboard Cite

BackgroundThe capability of large language models (LLMs) to understand and generate human-readable text has prompted the investigation of their potential as educational and management tools for cancer patients and healthcare providers.Materials and MethodsWe conducted a cross-sectional study aimed at evaluating the ability of ChatGPT-4, ChatGPT-3.5, and Google Bard to answer questions related to four domains of immuno-oncology (Mechanisms, Indications, Toxicities, and Prognosis). We generated 60 open-ended questions (15 for each section). Questions were manually submitted to LLMs, and responses were collected on June 30th, 2023. Two reviewers evaluated the answers independently.ResultsChatGPT-4 and ChatGPT-3.5 answered all questions, whereas Google Bard answered only 53.3% (p <0.0001). The number of questions with reproducible answers was higher for ChatGPT-4 (95%) and ChatGPT3.5 (88.3%) than for Google Bard (50%) (p <0.0001). In terms of accuracy, the number of answers deemed fully correct were 75.4%, 58.5%, and 43.8% for ChatGPT-4, ChatGPT-3.5, and Google Bard, respectively (p = 0.03). Furthermore, the number of responses deemed highly relevant was 71.9%, 77.4%, and 43.8% for ChatGPT-4, ChatGPT-3.5, and Google Bard, respectively (p = 0.04). Regarding readability, the number of highly readable was higher for ChatGPT-4 and ChatGPT-3.5 (98.1%) and (100%) compared to Google Bard (87.5%) (p = 0.02).ConclusionChatGPT-4 and ChatGPT-3.5 are potentially powerful tools in immuno-oncology, whereas Google Bard demonstrated relatively poorer performance. However, the risk of inaccuracy or incompleteness in the responses was evident in all three LLMs, highlighting the importance of expert-driven verification of the outputs returned by these technologies.IMPLICATIONS FOR PRACTICESeveral studies have recently evaluated whether large language models may be feasible tools for providing educational and management information for cancer patients and healthcare providers. In this cross-sectional study, we assessed the ability of ChatGPT-4, ChatGPT-3.5, and Google Bard to answer questions related to immuno-oncology. ChatGPT-4 and ChatGPT-3.5 returned a higher proportion of responses, which were more accurate and comprehensive, than those returned by Google Bard, yielding highly reproducible and readable outputs. These data support ChatGPT-4 and ChatGPT-3.5 as powerful tools in providing information on immuno-oncology; however, accuracy remains a concern, with expert assessment of the output still indicated.

show abstract

Section: Discussionmentioning

confidence: 99%

Comparison of Large Language Models in Answering Immuno-Oncology Questions: A Cross-Sectional Study

Iannantuono,

Bracken-Clarke,

Karzai

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently published studies suggest that ChatGPT outperforms Bard in answering lung cancer questions and neurosurgery oral boards preparation questions, which is consistent with our analysis. 9,10 ChatGPT-4 and Bard both display reasonable radiology knowledge. The role of AI in radiology spans many domains including image interpretation, tumour staging, report generation, workflow improvements, medical writing, and research.…”

Section: Discussionmentioning

confidence: 91%

Section: Discussionmentioning

confidence: 99%

“…7,8 Given the novelty of Bard by Google, to the best of our knowledge, there are only 2 published articles comparing ChatGPT to Bard, both of which found ChatGPT to be superior. 9,10 In this project, we aimed to compare the most recent version of ChatGPT, ChatGPT-4, and Bard in their ability to accurately respond to radiology board examination questions. Understanding the relative performance of these different chatbots can provide important insight into the strengths and weaknesses of each of these chatbots and which roles they are most suited to fill.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Comparative Performance of ChatGPT and Bard in a Text-Based Radiology Knowledge Assessment

et al. 2023

View full text Add to dashboard Cite

Purpose Bard by Google, a direct competitor to ChatGPT, was recently released. Understanding the relative performance of these different chatbots can provide important insight into their strengths and weaknesses as well as which roles they are most suited to fill. In this project, we aimed to compare the most recent version of ChatGPT, ChatGPT-4, and Bard by Google, in their ability to accurately respond to radiology board examination practice questions. Methods Text-based questions were collected from the 2017-2021 American College of Radiology’s Diagnostic Radiology In-Training (DXIT) examinations. ChatGPT-4 and Bard were queried, and their comparative accuracies, response lengths, and response times were documented. Subspecialty-specific performance was analyzed as well. Results 318 questions were included in our analysis. ChatGPT answered significantly more accurately than Bard (87.11% vs 70.44%, P < .0001). ChatGPT’s response length was significantly shorter than Bard’s (935.28 ± 440.88 characters vs 1437.52 ± 415.91 characters, P < .0001). ChatGPT’s response time was significantly longer than Bard’s (26.79 ± 3.27 seconds vs 7.55 ± 1.88 seconds, P < .0001). ChatGPT performed superiorly to Bard in neuroradiology, (100.00% vs 86.21%, P = .03), general & physics (85.39% vs 68.54%, P < .001), nuclear medicine (80.00% vs 56.67%, P < .01), pediatric radiology (93.75% vs 68.75%, P = .03), and ultrasound (100.00% vs 63.64%, P < .001). In the remaining subspecialties, there were no significant differences between ChatGPT and Bard’s performance. Conclusion ChatGPT displayed superior radiology knowledge compared to Bard. While both chatbots display reasonable radiology knowledge, they should be used with conscious knowledge of their limitations and fallibility. Both chatbots provided incorrect or illogical answer explanations and did not always address the educational content of the question.

show abstract

“…In radiology, ChatGPT and other LLMs have shown promising and innovative applications 2 . These include assistance with medical writing 3 and research 4 , structuring and organizing radiology reports 5–7 , protocoling radiology exams 8 , providing recommendations for screening 9,10 , answering patient questions 10,11 , taking a text-based radiology board style examination 12,13 , providing impressions 14 , assigning follow-up imaging according to established guidelines 15 among others.…”

Section: Introductionmentioning

confidence: 99%

Assessing Performance of Multimodal ChatGPT-4 on an image based Radiology Board-style Examination: An exploratory study

Bera,

Gupta,

Jiang

et al. 2024

Preprint

View full text Add to dashboard Cite

Objective To evaluate the performance of multimodal ChatGPT 4 on a radiology board-style examination containing text and radiologic images. Methods In this prospective exploratory study from October 30 to December 10, 2023, 110 multiple-choice questions containing images designed to match the style and content of radiology board examination like the American Board of Radiology Core or Canadian Board of Radiology examination were prompted to multimodal ChatGPT 4. Questions were further sub stratified according to lower-order (recall, understanding) and higher-order (analyze, synthesize), domains (according to radiology subspecialty), imaging modalities and difficulty (rated by both radiologists and radiologists-in-training). ChatGPT performance was assessed overall as well as in subcategories using Fisher exact test with multiple comparisons. Confidence in answering questions was assessed using a Likert scale (1-5) by consensus between a radiologist and radiologist-in-training. Reproducibility was assessed by comparing two different runs using two different accounts. Results ChatGPT 4 answered 55% (61/110) of image-rich questions correctly. While there was no significant difference in performance amongst the various sub-groups on exploratory analysis, performance was better on lower-order [61% (25/41)] when compared to higher-order [52% (36/69)] [P=.46]. Among clinical domains, performance was best on cardiovascular imaging [80% (8/10)], and worst on thoracic imaging [30% [3/10)]. Confidence in answering questions was confident/highly confident [89%(98/110)], even when incorrect There was poor reproducibility between two runs, with the answers being different in 14% (15/110) questions. Conclusion Despite no radiology specific pre-training, multimodal capabilities of ChatGPT appear promising on questions containing images. However, the lack of reproducibility among two runs, even with the same questions poses challenges of reliability.

show abstract

How AI Responds to Common Lung Cancer Questions: ChatGPT versus Google Bard

Cited by 162 publications

References 8 publications

Comparison of Large Language Models in Answering Immuno-Oncology Questions: A Cross-Sectional Study

Comparison of Large Language Models in Answering Immuno-Oncology Questions: A Cross-Sectional Study

Comparative Performance of ChatGPT and Bard in a Text-Based Radiology Knowledge Assessment

Assessing Performance of Multimodal ChatGPT-4 on an image based Radiology Board-style Examination: An exploratory study

Contact Info

Product

Resources

About