Large Language Model−Based Chatbot vs Surgeon-Generated Informed Consent Documentation for Common Procedures

Decker, Hannah; Trang, Karen; Ramirez, Joel; Colley, Alexis; Pierce, Logan; Coleman, Melissa; Bongiovanni, Tasce; Melton, Genevieve B.; Wick, Elizabeth

doi:10.1001/jamanetworkopen.2023.36997

Cited by 52 publications

(19 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The authors were primarily affiliated with institutions in the United States (n=47 of 122 different countries identified per publication, 38.5%), followed by Germany (n=11/122, 9%), Turkey (n=7/122, 5.7%), the United Kingdom (n=6/122, 4.9%), China/Australia/Italy (n=5/122, 4.1%, respectively), and 24 (n=36/122, 29.5%) other countries. Most studies examined one or more applications based on the GPT-3.5 architecture (n=66 of 124 different LLMs examined per study, 53.2%) 13,26–29,31–34,36–40,42–49,52–54,56–61,63,65–67,71,72,74,75,77,78,81–89,91,92,94,95,97–100,102–104,106–109,111 , followed by GPT-4 (n=33/124, 26.6%) 13,25,27,29,30,34–36,41,43,50,51,54,55,58,61,64,68–70,74,76,79–81,83,87,89,90,93,96,98,99,101,105 , Bard (n=10/124, 8.1%; now known as Gemini) 33,48,49,55,73,74,80,87,94,99 , Bing Chat (n=7/124, 5.7%; now Microsoft Copilot) 49,51,55,73,94,99,110 , and other applications based on Bidirectional Encoder Representations from Transformers (BERT; n=4/124, 3...…”

Section: Resultsmentioning

confidence: 99%

“…Most studies examined one or more applications based on the GPT-3.5 architecture (n=66 of 124 different LLMs examined per study, 53.2%) 13,26–29,31–34,36–40,42–49,52–54,56–61,63,65–67,71,72,74,75,77,78,81–89,91,92,94,95,97–100,102–104,106–109,111 , followed by GPT-4 (n=33/124, 26.6%) 13,25,27,29,30,34–36,41,43,50,51,54,55,58,61,64,68–70,74,76,79–81,83,87,89,90,93,96,98,99,101,105 , Bard (n=10/124, 8.1%; now known as Gemini) 33,48,49,55,73,74,80,87,94,99 , Bing Chat (n=7/124, 5.7%; now Microsoft Copilot) 49,51,55,73,94,99,110 , and other applications based on Bidirectional Encoder Representations from Transformers (BERT; n=4/124, 3.2%) 13,83,84 , Large Language Model Meta-AI (LLaMA; n=3/124, 2.4%) 55 , or Claude by Anthropic (n=1/124, 0.8%) 55 . The majority of applications were p...…”

Section: Resultsmentioning

confidence: 99%

Section: Characteristics Of Included Studiesmentioning

confidence: 99%

“…Most studies examined one or more applications based on the GPT-3.5 architecture (n=66 of 124 different LLMs examined per study, 53.2%) 13,[26][27][28][29][31][32][33][34][36][37][38][39][40][42][43][44][45][46][47][48][49][52][53][54][56][57][58][59][60][61]63,[65][66][67]71,72,74,75,77,78,[81][82][83][84][85][86][87][88][89]91,92,…”

Section: Characteristics Of Included Studiesmentioning

confidence: 99%

“…An overview of the presence of codes for each study is provided in Supplementary Section 3. The majority of articles investigated the use and feasibility of LLMs as medical chatbots (n=84/89, 94.4%) 13,[64][65][66]68,69,[98][99][100][101][102][103][104][105][106][107][108][109][110][111] , while fewer reports additionally or exclusively focused on the generation of patient information (n=19/89, 21.4%) 24,31,43,48,49,57,59,62,67,70,79,[88][89][90][91]97,102,106,107 , including clinical documentation such as informed consent forms (n=5/89, 5.6%) 43,67,91,97,102 and discharge instructions (n=1/89, 1.1%) 31 , or translation/summarization tasks of medical texts (n=5/89, 5.6%) 24,49,…”

Section: Applications Of Large Language Modelsmentioning

confidence: 99%

See 4 more Smart Citations

Systematic Review of Large Language Models for Patient Care: Current Applications and Challenges

Busch,

Hoffmann,

Rueger

et al. 2024

Preprint

View full text Add to dashboard Cite

The introduction of large language models (LLMs) into clinical practice promises to improve patient education and empowerment, thereby personalizing medical care and broadening access to medical knowledge. Despite the popularity of LLMs, there is a significant gap in systematized information on their use in patient care. Therefore, this systematic review aims to synthesize current applications and limitations of LLMs in patient care using a data-driven convergent synthesis approach. We searched 5 databases for qualitative, quantitative, and mixed methods articles on LLMs in patient care published between 2022 and 2023. From 4,349 initial records, 89 studies across 29 medical specialties were included, primarily examining models based on the GPT-3.5 (53.2%, n=66 of 124 different LLMs examined per study) and GPT-4 (26.6%, n=33/124) architectures in medical question answering, followed by patient information generation, including medical text summarization or translation, and clinical documentation. Our analysis delineates two primary domains of LLM limitations: design and output. Design limitations included 6 second-order and 12 third-order codes, such as lack of medical domain optimization, data transparency, and accessibility issues, while output limitations included 9 second-order and 32 third-order codes, for example, non-reproducibility, non-comprehensiveness, incorrectness, unsafety, and bias. In conclusion, this study is the first review to systematically map LLM applications and limitations in patient care, providing a foundational framework and taxonomy for their implementation and evaluation in healthcare settings.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Characteristics Of Included Studiesmentioning

confidence: 99%

Section: Characteristics Of Included Studiesmentioning

confidence: 99%

Section: Applications Of Large Language Modelsmentioning

confidence: 99%

See 3 more Smart Citations

Systematic Review of Large Language Models for Patient Care: Current Applications and Challenges

Busch,

Hoffmann,

Rueger

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

Development and Evaluation of a Retrieval-Augmented Large Language Model Framework for Ophthalmology

Luo,

Pang,

et al. 2024

JAMA Ophthalmol

View full text Add to dashboard Cite

ImportanceAlthough augmenting large language models (LLMs) with knowledge bases may improve medical domain–specific performance, practical methods are needed for local implementation of LLMs that address privacy concerns and enhance accessibility for health care professionals.ObjectiveTo develop an accurate, cost-effective local implementation of an LLM to mitigate privacy concerns and support their practical deployment in health care settings.Design, Setting, and ParticipantsChatZOC (Sun Yat-Sen University Zhongshan Ophthalmology Center), a retrieval-augmented LLM framework, was developed by enhancing a baseline LLM with a comprehensive ophthalmic dataset and evaluation framework (CODE), which includes over 30 000 pieces of ophthalmic knowledge. This LLM was benchmarked against 10 representative LLMs, including GPT-4 and GPT-3.5 Turbo (OpenAI), across 300 clinical questions in ophthalmology. The evaluation, involving a panel of medical experts and biomedical researchers, focused on accuracy, utility, and safety. A double-masked approach was used to try to minimize bias assessment across all models. The study used a comprehensive knowledge base derived from ophthalmic clinical practice, without directly involving clinical patients.ExposuresLLM response to clinical questions.Main Outcomes and MeasuresAccuracy, utility, and safety of LLMs in responding to clinical questions.ResultsThe baseline model achieved a human ranking score of 0.48. The retrieval-augmented LLM had a score of 0.60, a difference of 0.12 (95% CI, 0.02-0.22; P = .02) from baseline and not different from GPT-4 with a score of 0.61 (difference = 0.01; 95% CI, −0.11 to 0.13; P = .89). For scientific consensus, the retrieval-augmented LLM was 84.0% compared with the baseline model of 46.5% (difference = 37.5%; 95% CI, 29.0%-46.0%; P &lt; .001) and not different from GPT-4 with a value of 79.2% (difference = 4.8%; 95% CI, −0.3% to 10.0%; P = .06).Conclusions and RelevanceResults of this quality improvement study suggest that the integration of high-quality knowledge bases improved the LLM’s performance in medical domains. This study highlights the transformative potential of augmented LLMs in clinical practice by providing reliable, safe, and practical clinical information. Further research is needed to explore the broader application of such frameworks in the real world.

show abstract

Uses and limitations of artificial intelligence for oncology

Kolla,

Parikh

2024

Cancer

View full text Add to dashboard Cite

Modern artificial intelligence (AI) tools built on high‐dimensional patient data are reshaping oncology care, helping to improve goal‐concordant care, decrease cancer mortality rates, and increase workflow efficiency and scope of care. However, data‐related concerns and human biases that seep into algorithms during development and post‐deployment phases affect performance in real‐world settings, limiting the utility and safety of AI technology in oncology clinics. To this end, the authors review the current potential and limitations of predictive AI for cancer diagnosis and prognostication as well as of generative AI, specifically modern chatbots, which interfaces with patients and clinicians. They conclude the review with a discussion on ongoing challenges and regulatory opportunities in the field.

show abstract

Large Language Model−Based Chatbot vs Surgeon-Generated Informed Consent Documentation for Common Procedures

Cited by 52 publications

References 36 publications

Systematic Review of Large Language Models for Patient Care: Current Applications and Challenges

Systematic Review of Large Language Models for Patient Care: Current Applications and Challenges

Development and Evaluation of a Retrieval-Augmented Large Language Model Framework for Ophthalmology

Uses and limitations of artificial intelligence for oncology

Contact Info

Product

Resources

About