A Survey of Clinicians' Views of the Utility of Large Language Models

Abstract

Objective: Large language models (LLMs) like ChatGPT are powerful algorithms that have been shown to produce human-like text from input data. A number of potential clinical applications of this technology have been proposed and evaluated by biomedical informatics experts. However, few have surveyed healthcare providers for their opinions about whether the technology is fit for use. Materials and Methods: We distributed a validated mixed-methods survey to gauge practicing clinicians’ comfort with LLMs for a breadth of tasks in clinical practice, research and education, which were selected from the literature. Results: A total of 30 clinicians fully completed the survey. Of the 23 tasks, 16 were rated positively by more than 50% of the respondents. Based on our qualitative analysis, healthcare providers considered LLMs to have excellent synthesis skills and efficiency. However, our respondents had concerns that LLMs could generate false information and propagate training data bias. Discussion: Our survey respondents were most comfortable with scenarios that allow LLMs to function in an assistive role, like a physician extender or trainee. Conclusion: In a uniquely rigorous and comprehensive mixed-methods survey of clinicians about LLM use, healthcare providers were encouraging of having LLMs in healthcare for many tasks, and especially in assistive roles. There is a need for continued human-centered development of both LLMs and artificial intelligence (AI) in general.

Clinician Perceptions of Generative Artificial Intelligence Tools and Clinical Workflows: Potential Uses, Motivations for Adoption, and Sentiments on Impact

“…Furthermore, integration of generative AI into healthcare is a complex endeavor, requiring stakeholders such as researchers, vendors, health systems, and frontline clinicians to make strategic choices amidst time, resource, and expertise constraints. Prior studies have focused on the use of a generative AI tool to address a specific clinical task, such as answering patient questions or generating discharge summaries, but there has been limited exploration into systematically explore clinicians' views on which specific types of clinical tasks would be most suitable for a generative AI intervention 14 . Additionally, given that the performance of many generative AI models rely on prompt quality, if clinicians are expected to design these inputs, there is a need to assess clinicians' understanding and ability to generate prompts that would yield useful, accurate and relevant outputs 15 .…”