Large language modeling and classical AI methods for the future of healthcare

Banerjee, Sri; Dunn, Pat; Conard, Scott; Ng, Roger

doi:10.1016/j.glmedi.2023.100026

Cited by 7 publications

(1 citation statement)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This study investigated the impact of embedding a GPT model (RAG) on its performance and its comparison with ChatGPT with a keen focus on response accuracy, clarity, relevance, up-to-date knowledge. Accuracy was assessed because inaccurate information can lead to misunderstandings by patients, potentially impacting their recovery [ 18 ]. Evaluating the relevance ensured that the model-generated responses are directly applicable to the patient’s situation.…”

Section: Discussionmentioning

confidence: 99%

Leveraging Large Language Models in the delivery of post-operative dental care: a comparison between an embedded GPT model and ChatGPT

Batool,

Naved,

Kazmi

et al. 2024

BDJ Open

View full text Add to dashboard Cite

Objective This study underscores the transformative role of Artificial Intelligence (AI) in healthcare, particularly the promising applications of Large Language Models (LLMs) in the delivery of post-operative dental care. The aim is to evaluate the performance of an embedded GPT model and its comparison with ChatGPT-3.5 turbo. The assessment focuses on aspects like response accuracy, clarity, relevance, and up-to-date knowledge in addressing patient concerns and facilitating informed decision-making. Material and methods An embedded GPT model, employing GPT-3.5-16k, was crafted via GPT-trainer to answer postoperative questions in four dental specialties including Operative Dentistry & Endodontics, Periodontics, Oral & Maxillofacial Surgery, and Prosthodontics. The generated responses were validated by thirty-six dental experts, nine from each specialty, employing a Likert scale, providing comprehensive insights into the embedded GPT model’s performance and its comparison with GPT3.5 turbo. For content validation, a quantitative Content Validity Index (CVI) was used. The CVI was calculated both at the item level (I-CVI) and scale level (S-CVI/Ave). To adjust I-CVI for chance agreement, a modified kappa statistic (K*) was computed. Results The overall content validity of responses generated via embedded GPT model and ChatGPT was 65.62% and 61.87% respectively. Moreover, the embedded GPT model revealed a superior performance surpassing ChatGPT with an accuracy of 62.5% and clarity of 72.5%. In contrast, the responses generated via ChatGPT achieved slightly lower scores, with an accuracy of 52.5% and clarity of 67.5%. However, both models performed equally well in terms of relevance and up-to-date knowledge. Conclusion In conclusion, embedded GPT model showed better results as compared to ChatGPT in providing post-operative dental care emphasizing the benefits of embedding and prompt engineering, paving the way for future advancements in healthcare applications.

show abstract

Section: Discussionmentioning

confidence: 99%