Generating medically-accurate summaries of patient-provider dialogue: A multi-stage approach using large language models

Nair, Varun Sasidharan; Schumacher, Elliot; Kannan, Anitha

doi:10.18653/v1/2023.clinicalnlp-1.26

Cited by 3 publications

(2 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It is important to stress that our evaluation of the GPT-4 model was based on single queries using prompts developed by informal experimentation. Further research should investigate more sophisticated prompting strategies such as "chain of thought" 36 for gene set analyses and the use of methods that apply external tools and orchestrate multiple LLM interactions [37][38][39][40][41][42][43] , such as integrating literature searches into the LLM analysis rather than as a post hoc verification method. Our study also makes the implicit assumption that all gene sets emerging from a biological study have a coherent function to be summarized.…”

Section: Discussionmentioning

confidence: 99%

Evaluation of large language models for discovery of gene set function

Pratt,

Hu,

Alkhairy

et al. 2023

Preprint

View full text Add to dashboard Cite

Gene set analysis is a mainstay of functional genomics, but it relies on manually curated databases of gene functions that are incomplete and unaware of biological context. Here we evaluate the ability of OpenAI’s GPT-4, a Large Language Model (LLM), to develop hypotheses about common gene functions from its embedded biomedical knowledge. We created a GPT-4 pipeline to label gene sets with names that summarize their consensus functions, substantiated by analysis text and citations. Benchmarking against named gene sets in the Gene Ontology, GPT-4 generated very similar names in 50% of cases, while in most remaining cases it recovered the name of a more general concept. In gene sets discovered in ‘omics data, GPT-4 names were more informative than gene set enrichment, with supporting statements and citations that largely verified in human review. The ability to rapidly synthesize common gene functions positions LLMs as valuable functional genomics assistants.

show abstract

Section: Discussionmentioning

confidence: 99%

Evaluation of large language models for discovery of gene set function

Pratt,

Hu,

Alkhairy

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…The predominant method for evaluating LLMs in the medical field involves medical exam-type questions, with a strong emphasis on multiple-choice formats [16][17][18] . Although there are instances where LLMs are tested on free-response and reasoning tasks 19,20,12 , or for medical conversation summarization and care plan generation 21 , these are less common. However, these assessments do not explore LLMs' ability for engaging in interactive patient conversations, a crucial aspect of their potential role in revolutionizing healthcare delivery.…”

Section: Introductionmentioning

confidence: 99%

Guidelines For Rigorous Evaluation of Clinical LLMs For Conversational Reasoning

Johri,

Jeong,

Tran

et al. 2023

Preprint

View full text Add to dashboard Cite

Large Language Models (LLMs) show promise for medical diagnosis, but traditional evaluations using static exam questions overlook the complexity of real-world clinical dialogues. We introduce a multi-agent conversational framework where doctor-AI and patient-AI agents interact to diagnose medical conditions, evaluated by a grader-AI agent and medical experts. We assessed the diagnostic accuracy of GPT-4 and GPT-3.5, in conversational versus static settings using 140 cases focusing on skin diseases. Our study revealed a decline in diagnostic accuracy, unmasking key limitations in LLMs’ ability to integrate details from conversational interactions to improve diagnostic performance. We introduced Conversational Summarization, a technique that enhanced performance, and expert review identified deficiencies compared to human dermatologists in comprehensive history gathering, appropriate use of terminology, and reliability. Our findings advocate for nuanced, rigorous evaluation of LLMs before clinical integration, and our framework represents a significant advancement toward responsible testing methodologies in medicine.

show abstract

Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults

Yang,

Xu,

Yao

et al. 2024

Proc. ACM Interact. Mob. Wearable Ubiquitous Technol.

View full text Add to dashboard Cite

Despite the plethora of telehealth applications to assist home-based older adults and healthcare providers, basic messaging and phone calls are still the most common communication methods, which suffer from limited availability, information loss, and process inefficiencies. One promising solution to facilitate patient-provider communication is to leverage large language models (LLMs) with their powerful natural conversation and summarization capability. However, there is a limited understanding of LLMs' role during the communication. We first conducted two interview studies with both older adults (N=10) and healthcare providers (N=9) to understand their needs and opportunities for LLMs in patient-provider asynchronous communication. Based on the insights, we built an LLM-powered communication system, Talk2Care, and designed interactive components for both groups: (1) For older adults, we leveraged the convenience and accessibility of voice assistants (VAs) and built an LLM-powered conversational interface for effective information collection. (2) For health providers, we built an LLM-based dashboard to summarize and present important health information based on older adults' conversations with the VA. We further conducted two user studies with older adults and providers to evaluate the usability of the system. The results showed that Talk2Care could facilitate the communication process, enrich the health information collected from older adults, and considerably save providers' efforts and time. We envision our work as an initial exploration of LLMs' capability in the intersection of healthcare and interpersonal communication.

show abstract

Generating medically-accurate summaries of patient-provider dialogue: A multi-stage approach using large language models

Cited by 3 publications

References 0 publications

Evaluation of large language models for discovery of gene set function

Evaluation of large language models for discovery of gene set function

Guidelines For Rigorous Evaluation of Clinical LLMs For Conversational Reasoning

Talk2Care: An LLM-based Voice Assistant for Communication between Healthcare Providers and Older Adults

Contact Info

Product

Resources

About