Evaluating ChatGPT as an Adjunct for Radiologic Decision-Making

Rao, Arya; Kim, John; Kamineni, Meghana; Pang, Michael; Lie, Winston; Succi, Marc D.

doi:10.1101/2023.02.02.23285399

Cited by 169 publications

(172 citation statements)

References 21 publications

Supporting

Mentioning

168

Contrasting

Unclassified

Order By: Relevance

“…For breast cancer, there was a positive correlation between severity and accuracy, and for breast pain there was a negative correlation. 17 Given that the data in this study covers 36 different clinical scenarios as opposed to trends within specific clinical conditions, we suspect that any association between acuity of presentation and accuracy could be found on a within-case basis, as opposed to between cases.…”

Section: Discussionmentioning

confidence: 92%

“…The latter observation is in line with Rao et al's observation that ChatGPT struggles to identify situations in which diagnostic testing is futile. 17 Resource utilization was not explicitly tested in our study; further prompt engineering could be performed to evaluate ChatGPT's ability to recommend the appropriate utilization of resources (for example, asking "What tests are appropriate clinically while also taking cost management into account? ").…”

Section: Discussionmentioning

confidence: 99%

“…15 Recently, there has been great interest in utilizing the nascent but powerful chatbot for clinical decision support. [16][17][18] Given that LLMs like ChatGPT have the ability to integrate large amounts of textual information to synthesize responses to human-generated prompts, we speculated that ChatGPT would be able to act as an on-the-ground copilot in clinical reasoning, making use of the wealth of information available during patient care from the Electronic Health Record (EHR) and other sources. We focused on comprehensive clinical vignettes as a model, and tested the hypothesis that when provided clinical vignettes, ChatGPT would be able to recommend diagnostic workup, decide the clinical management course, and ultimately make the diagnosis, working through the entire clinical encounter.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow

Rao

Pang

Kim

et al. 2023

Preprint

Self Cite

111

View full text Add to dashboard Cite

IMPORTANCE: Large language model (LLM) artificial intelligence (AI) chatbots direct the power of large training datasets towards successive, related tasks, as opposed to single-ask tasks, for which AI already achieves impressive performance. The capacity of LLMs to assist in the full scope of iterative clinical reasoning via successive prompting, in effect acting as virtual physicians, has not yet been evaluated. OBJECTIVE: To evaluate ChatGPT′s capacity for ongoing clinical decision support via its performance on standardized clinical vignettes. DESIGN: We inputted all 36 published clinical vignettes from the Merck Sharpe & Dohme (MSD) Clinical Manual into ChatGPT and compared accuracy on differential diagnoses, diagnostic testing, final diagnosis, and management based on patient age, gender, and case acuity. SETTING: ChatGPT, a publicly available LLM PARTICIPANTS: Clinical vignettes featured hypothetical patients with a variety of age and gender identities, and a range of Emergency Severity Indices (ESIs) based on initial clinical presentation. EXPOSURES: MSD Clinical Manual vignettes MAIN OUTCOMES AND MEASURES: We measured the proportion of correct responses to the questions posed within the clinical vignettes tested. RESULTS: ChatGPT achieved 71.7% (95% CI, 69.3% to 74.1%) accuracy overall across all 36 clinical vignettes. The LLM demonstrated the highest performance in making a final diagnosis with an accuracy of 76.9% (95% CI, 67.8% to 86.1%), and the lowest performance in generating an initial differential diagnosis with an accuracy of 60.3% (95% CI, 54.2% to 66.6%). Compared to answering questions about general medical knowledge, ChatGPT demonstrated inferior performance on differential diagnosis (β=-15.8%, p<0.001) and clinical management (β=-7.4%, p=0.02) type questions. CONCLUSIONS AND RELEVANCE: ChatGPT achieves impressive accuracy in clinical decision making, with particular strengths emerging as it has more clinical information at its disposal.

show abstract

Section: Discussionmentioning

confidence: 92%

Section: Discussionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Assessing the Utility of ChatGPT Throughout the Entire Clinical Workflow

Rao

Pang

Kim

et al. 2023

Preprint

Self Cite

111

View full text Add to dashboard Cite

show abstract

“…[15] Due to the strong writing ability, ChatGPT has been shown to capable of writing sophisticated and seemingly essays and talks, summarize literature and even generate computer code. [16] The use of ChatGPT also has been successfully practiced by scientist across the globe [17][18][19] and we believe that its widespread to all fields is unstoppable. Therefore, it is curious for the authors to know that how can ChatGPT benefit the field of pharmacy.…”

Section: Ofmentioning

confidence: 99%

How Can ChatGPT Benefit Pharmacy: A Case Report on Review Writing

Zhu¹,

Han²,

Chen³

et al. 2023

Preprint

View full text Add to dashboard Cite

Artificial Intelligence (AI) is a breakthrough&ensp;technology that has been widely applied in many fields and its use in pharmacy is also gaining increasing attentions. Recently, ChatGPT, a newly developed virtual assistant and large language model, is showing strong ability in writing, which makes it a potential tool for review writing. Here in our study, we employ the public available ChatGPT and try to instruct it to generate a mini review on the topic of “lipid-based drug delivery systems” by conversation. Our results showed that ChatGPT can give reasonable outline for this topic and give seemingly sentences with concluding wording, which can give useful information to the readers. However, the accuracy and consistence of the review are not fully guaranteed and there is lack of reliable citations. Therefore, it is concluded that ChatGPT can only give general knowledge on the selected topic, but not capable of giving in-depth discussion. The generated review article can be a useful but occasionally unprecise electronic encyclopedia instead of rigorous scientific literature, which can give benefit for pharmaceutical education and interchange to some extent.

show abstract

“… 10 In addition, AI technologies play an important role in disease prediction, diagnosis and assessment of therapeutic targets, such as providing treatment guidelines for cancer patients based on their magnetic resonance imaging radiomics and predicting ageing‐related diseases. 11 , 12 , 13 However, unlike the AI algorithm or model specially developed for drug discovery and disease diagnosis, the core value and advantage of ChatGPT lies in its powerful LLM. At present, ChatGPT cannot update the training data in a real‐time manner.…”

mentioning

confidence: 99%