Developing prompts from large language model for extracting clinical information from pathology and ultrasound reports in breast cancer

Abstract

Purpose: We aimed to evaluate the time and cost of developing prompts using large language model (LLM), tailored to extract clinical factors in breast cancer patients and their accuracy.Materials and Methods: We collected data from reports of surgical pathology and ultrasound from breast cancer patients who underwent radiotherapy from 2020 to 2022. We extracted the information using the Generative Pre-trained Transformer (GPT) for Sheets and Docs extension plugin and termed this the “LLM” method. The time and cost of developing the prompts with LLM methods were assessed and compared with those spent on collecting information with “full manual” and “LLM-assisted manual” methods. To assess accuracy, 340 patients were randomly selected, and the extracted information by LLM method were compared with those collected by “full manual” method.Results: Data from 2,931 patients were collected. We developed 12 prompts for Extract function and 12 for Format function to extract and standardize the information. The overall accuracy was 87.7%. For lymphovascular invasion, it was 98.2%. Developing and processing the prompts took 3.5 hours and 15 minutes, respectively. Utilizing the ChatGPT application programming interface cost US $65.8 and when factoring in the estimated wage, the total cost was US $95.4. In an estimated comparison, “LLM-assisted manual” and “LLM” methods were time- and cost-efficient compared to the “full manual” method.Conclusion: Developing and facilitating prompts for LLM to derive clinical factors was efficient to extract crucial information from huge medical records. This study demonstrated the potential of the application of natural language processing using LLM model in breast cancer patients. Prompts from the current study can be re-used for other research to collect clinical information.

Applications of Large Language Models (LLMs) in Breast Cancer Care

“…Rao et al and Haver et al evaluated LLMs for breast imaging recommendations 16,17 , Sorin et al and Lukac et al evaluated LLMs as supportive decision making tools in multidisciplinary tumor boards 13,15 , and Choi et al used LLM for information extraction from ultrasound and pathology reports 14 , ( Figure 2 ). Performance of LLMs on different applications ranged from 64-98%.…”

“…Performance of LLMs on different applications ranged from 64-98%. Best performance rates were achieved for information extraction and question-answering, with correct responses ranging from 88-98% 14,16 ( Table 2 ).…”

“…Sorin et al noticed absolute lack of referral to imaging 13 , while Rao et al who evaluated appropriateness of imaging noticed imaging overutilization 16 . Some of the studies also discussed prompt sensitivity 14,17 , and difficulty to verify the reliability of the answers 15-17 .…”

A Systematic Review of Testing and Evaluation of Healthcare Applications of Large Language Models (LLMs)

“…Our recent JAMA special communication pointed out that testing LLMs with hypothetical medical questions is like assessing a car’s performance with multiple-choice questions before certifying it for road use 51 . Real patient care data encompasses the complexities of clinical practice, providing a more thorough evaluation of LLM performance that will closely mirror real-world performance 52 53 54 55 .…”

The Transformative Potential of Large Language Models in Mining Electronic Health Records Data

“…Focusing on the significance of appropriate instructions (prompts), researchers such as Hyeon Seok Choi et al [8] highlighted that the gpt-3.5-turbo model exhibited an accuracy rate of 87.7% in extracting information from pathology and ultrasound reports of breast cancer patients. This achievement represents a notable advancement over traditional natural language processing models.…”