Head‐to‐Head Comparison of ChatGPT Versus Google Search for Medical Knowledge Acquisition

Ayoub, Noel F.; Lee, Yu‐Jin; Grimm, David; Divi, Vasu

doi:10.1002/ohn.465

Cited by 42 publications

(17 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…GPT responses have been noted to be as reliable as web searches and are closer to clinical guidelines, making it a promising tool to support medical communication. 7 8 In this study, the GPT returned useful results comparable to the guidelines, not calling for excessive pessimism or optimism. However, GPT-based questions and answers tended to return counselling over pharmacological treatment options, with many responses encouraging consultation with medical staff.…”

Section: Discussionsupporting

confidence: 53%

“…When disseminating information about cancer treatment and sexual health issues faced by cancer survivors, the generated chatbots functioned without refusing to answer, with or without training sources of medical guidelines. GPT responses have been noted to be as reliable as web searches and are closer to clinical guidelines, making it a promising tool to support medical communication 7 8. In this study, the GPT returned useful results comparable to the guidelines, not calling for excessive pessimism or optimism.…”

Section: Discussionmentioning

confidence: 63%

See 1 more Smart Citation

Generative artificial intelligence and non-pharmacological bias: an experimental study on cancer patient sexual health communications

Hanai,

Ishikawa,

Kawauchi

et al. 2024

BMJ Health Care Inform

View full text Add to dashboard Cite

ObjectivesThe objective of this study was to explore the feature of generative artificial intelligence (AI) in asking sexual health among cancer survivors, which are often challenging for patients to discuss.MethodsWe employed the Generative Pre-trained Transformer-3.5 (GPT) as the generative AI platform and used DocsBot for citation retrieval (June 2023). A structured prompt was devised to generate 100 questions from the AI, based on epidemiological survey data regarding sexual difficulties among cancer survivors. These questions were submitted to Bot1 (standard GPT) and Bot2 (sourced from two clinical guidelines).ResultsNo censorship of sexual expressions or medical terms occurred. Despite the lack of reflection on guideline recommendations, ‘consultation’ was significantly more prevalent in both bots’ responses compared with pharmacological interventions, with ORs of 47.3 (p<0.001) in Bot1 and 97.2 (p<0.001) in Bot2.DiscussionGenerative AI can serve to provide health information on sensitive topics such as sexual health, despite the potential for policy-restricted content. Responses were biased towards non-pharmacological interventions, which is probably due to a GPT model designed with the ’s prohibition policy on replying to medical topics. This shift warrants attention as it could potentially trigger patients’ expectations for non-pharmacological interventions.

show abstract

Section: Discussionsupporting

confidence: 53%

Section: Discussionmentioning

confidence: 63%

Generative artificial intelligence and non-pharmacological bias: an experimental study on cancer patient sexual health communications

Hanai,

Ishikawa,

Kawauchi

et al. 2024

BMJ Health Care Inform

View full text Add to dashboard Cite

show abstract

“…Given that models like ChatGPT are already publicly accessible and widely used, patients may already refer to them for medical questions in much the same way they use Google Search, making concerns about their early adoption somewhat academic. 125…”

Section: Discussionmentioning

confidence: 99%

Systematic Review of Large Language Models for Patient Care: Current Applications and Challenges

Busch,

Hoffmann,

Rueger

et al. 2024

Preprint

View full text Add to dashboard Cite

The introduction of large language models (LLMs) into clinical practice promises to improve patient education and empowerment, thereby personalizing medical care and broadening access to medical knowledge. Despite the popularity of LLMs, there is a significant gap in systematized information on their use in patient care. Therefore, this systematic review aims to synthesize current applications and limitations of LLMs in patient care using a data-driven convergent synthesis approach. We searched 5 databases for qualitative, quantitative, and mixed methods articles on LLMs in patient care published between 2022 and 2023. From 4,349 initial records, 89 studies across 29 medical specialties were included, primarily examining models based on the GPT-3.5 (53.2%, n=66 of 124 different LLMs examined per study) and GPT-4 (26.6%, n=33/124) architectures in medical question answering, followed by patient information generation, including medical text summarization or translation, and clinical documentation. Our analysis delineates two primary domains of LLM limitations: design and output. Design limitations included 6 second-order and 12 third-order codes, such as lack of medical domain optimization, data transparency, and accessibility issues, while output limitations included 9 second-order and 32 third-order codes, for example, non-reproducibility, non-comprehensiveness, incorrectness, unsafety, and bias. In conclusion, this study is the first review to systematically map LLM applications and limitations in patient care, providing a foundational framework and taxonomy for their implementation and evaluation in healthcare settings.

show abstract

“…Hence, LLM services will likely soon replace the traditional search engines as the primary source of medical information for lay people. 3 , 4 , 5 , 6 …”

Section: Introductionmentioning

confidence: 99%

“…Hence, LLM services will likely soon replace the traditional search engines as the primary source of medical information for lay people. [3][4][5][6] LLMs are powered by a 'deep neural network architecture' and leverage principles of natural language processing. These models employ unsupervised learning, where they analyse extensive text data to learn linguistic patterns, grammar rules and contextual nuances.…”

mentioning

confidence: 99%

Urology consultants versus large language models: Potentials and hazards for medical advice in urology

Eckrich,

Ellinger,

Cox

et al. 2024

BJUI Compass

View full text Add to dashboard Cite

BackgroundCurrent interest surrounding large language models (LLMs) will lead to an increase in their use for medical advice. Although LLMs offer huge potential, they also pose potential misinformation hazards.ObjectiveThis study evaluates three LLMs answering urology‐themed clinical case‐based questions by comparing the quality of answers to those provided by urology consultants.MethodsForty‐five case‐based questions were answered by consultants and LLMs (ChatGPT 3.5, ChatGPT 4, Bard). Answers were blindly rated using a six‐step Likert scale by four consultants in the categories: ‘medical adequacy’, ‘conciseness’, ‘coherence’ and ‘comprehensibility’. Possible misinformation hazards were identified; a modified Turing test was included, and the character count was matched.ResultsHigher ratings in every category were recorded for the consultants. LLMs' overall performance in language‐focused categories (coherence and comprehensibility) was relatively high. Medical adequacy was significantly poorer compared with the consultants. Possible misinformation hazards were identified in 2.8% to 18.9% of answers generated by LLMs compared with <1% of consultant's answers. Poorer conciseness rates and a higher character count were provided by LLMs. Among individual LLMs, ChatGPT 4 performed best in medical accuracy (p < 0.0001) and coherence (p = 0.001), whereas Bard received the lowest scores. Generated responses were accurately associated with their source with 98% accuracy in LLMs and 99% with consultants.ConclusionsThe quality of consultant answers was superior to LLMs in all categories. High semantic scores for LLM answers were found; however, the lack of medical accuracy led to potential misinformation hazards from LLM ‘consultations’. Further investigations are necessary for new generations.

show abstract

Head‐to‐Head Comparison of ChatGPT Versus Google Search for Medical Knowledge Acquisition

Cited by 42 publications

References 14 publications

Generative artificial intelligence and non-pharmacological bias: an experimental study on cancer patient sexual health communications

Generative artificial intelligence and non-pharmacological bias: an experimental study on cancer patient sexual health communications

Systematic Review of Large Language Models for Patient Care: Current Applications and Challenges

Urology consultants versus large language models: Potentials and hazards for medical advice in urology

Contact Info

Product

Resources

About