Reply to Moreno et al. Comment on “Sallam, M. ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns. Healthcare 2023, 11, 887”

Sallam, Malik

doi:10.3390/healthcare11222955

Cited by 8 publications

(5 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This limitation of generative AI performance is particularly relevant in healthcare education, where the ability to apply knowledge creatively and critically is essential [43]. The observed limitation raises concerns about the current reliability of AI as an educational tool, which was reported in the context of various AI chatbots [7,18,34,44]. Collectively, these results highlight the critical areas for future development and improvement in AI training approaches.…”

Section: Discussionmentioning

confidence: 98%

“…This variability can be attributed to different factors, such as the AI model used, the prompting approach, and importantly the language(s) used in prompting [31,32]. Such ndings highlight the necessity for continued research to elucidate the determinants of AI models' performance, thereby informing the re nement of AI algorithms for improved performance and subsequent improved utility in various disciplines such as healthcare education [7,33,34].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The Performance of OpenAI ChatGPT-4 and Google Gemini in Virology Multiple-Choice Questions: A Comparative Analysis of English and Arabic Responses

Sallam,

Al-Mahzoum,

Almutawaa

et al. 2024

Preprint

View full text Add to dashboard Cite

Background: The integration of artificial intelligence (AI) in healthcare education is inevitable. Understanding the proficiency of generative AI in different languages to answer complex questions is crucial for educational purposes. Objective: To compare the performance ChatGPT-4 and Gemini in answering Virology multiple-choice questions (MCQs) in English and Arabic, while assessing the quality of the generated content. Methods: Both AI models’ responses to 40 Virology MCQs were assessed for correctness and quality based on the CLEAR tool designed for evaluation of AI-generated content. The MCQs were classified into lower and higher cognitive categories based on the revised Bloom’s taxonomy. The study design considered the METRICS checklist for the design and reporting of generative AI-based studies in healthcare. Results: ChatGPT-4 and Gemini performed better in English compared to Arabic, with ChatGPT-4 consistently surpassing Gemini in correctness and CLEAR scores. ChatGPT-4 led Gemini with 80% vs. 62.5% correctness in English compared to 65% vs. 55% in Arabic. For both AI models, superior performance in lower cognitive domains was reported. Conclusion: Both ChatGPT-4 and Gemini exhibited potential in educational applications; nevertheless, their performance varied across languages highlighting the importance of continued development to ensure the effective AI integration in healthcare education globally.

show abstract

Section: Discussionmentioning

confidence: 98%

Section: Introductionmentioning

confidence: 99%

The Performance of OpenAI ChatGPT-4 and Google Gemini in Virology Multiple-Choice Questions: A Comparative Analysis of English and Arabic Responses

Sallam,

Al-Mahzoum,

Almutawaa

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…Even though general purpose huge LLMs have demonstrated amazing capabilities to generalize to a wide variety of domains as zero-shot frameworks, their performance may still suffer in terms of relevance or specificity to a sensitive domain and may “ hallucinate ” while being forced to generate critical information 14,16,25 . Training of these models require humongous text corpora usually curated from publicly available text context available on the world wide web.…”

Section: Discussionmentioning

confidence: 99%

“…They may generate generic responses which can be incompetent to domain-specific questions. They also demonstrate known issues of hallucination which can be especially dangerous in the sensitive domains like medicine 13–16 . Research effort has been put into domain-specific LLM development such as MedPALM 17 (540B), MedAlpaca 18 (13B), and BioGTP 19 (1.5B).…”

Section: Introductionmentioning

confidence: 92%

Domain-specific LLM Development and Evaluation – A Case-study for Prostate Cancer

Tariq,

Luo,

Urooj

et al. 2024

Preprint

View full text Add to dashboard Cite

In this work, we present our strategy for developing domain-specific large language models which cover the vocabulary of the target domain and train on reliable sources of clinical information. Prostate cancer was chosen as a use-case for this study. We collected more than 1.8 million clinical notes and radiology and pathology reports for 15341 patients treated for prostate cancer in Mayo Clinic across three sites and outpatient clinics. In addition to domain-specific training data, we built domain-specific tokenizers and devised knowledge-guided training strategies for LLM development. During the self-supervised training, LLM was forced to predict domain-specific information by marking clinical terms using UMLS parser. We evaluated the model for downstream tasks of clinical information prediction and question answering using quantitative and user evaluation study to measure the accuracy, reliability and information completeness. We compared the domain-specific model against similarly sized general purpose model GPT-2 and a three-times larger domain specialized model. i.e., BioGPT. Our model outperformed GPT-2 on both tasks by a wide margin. Our model was also able to outperform BioGPT on clinical information prediction tasks and showed some advantages over BioGPT in question-answering tasks.

show abstract

“…Similarly, the potential of models such as ChatGPT to enhance medical practices, while addressing concerns about safety, ethics, and maintaining the human element in patient care, has been reviewed in [7]. A related systematic review evaluated the benefits and limitations of ChatGPT in healthcare education, research, and practice, based on an analysis of 60 publications from PubMed/MEDLINE and Google Scholar [14]. Insights into the opportunities and challenges of LLMs for biomedical or clinical usage, including their roles in pre-consultation, diagnosis, data management, and their support for medical education and writing, have also been offered in [15].…”

Section: Introductionmentioning

confidence: 99%

Large Language Models to process, analyze, and synthesize biomedical texts – a scoping review

Doneva,

Qin,

Sick

et al. 2024

Preprint

View full text Add to dashboard Cite

The advent of large language models (LLMs) such as BERT and, more recently, GPT, is transforming our approach of analyzing and understanding biomedical texts. To stay informed about the latest advancements in this area, there is a need for up-to-date summaries on the impact of LLM on Natural Language Processing (NLP) in the biomedical field. Thus, this scoping review aims to provide a detailed overview of the current state of biomedical NLP research and its applications, with a special focus on the evolving role of LLMs. We conducted a systematic search of PubMed, EMBASE, and Google Scholar for studies and conference proceedings published from 2017 to December 19, 2023, that develop or utilize LLMs for NLP tasks in biomedicine. From 13,823 references, we selected 199 publications and conference proceedings for our review. LLMs are being applied to a wide array of tasks in the biomedical field, including knowledge management, text mining, drug discovery, and evidence synthesis. Prominent among these tasks are text classification, relation extraction, and named entity recognition. Although BERT-based models remain prevalent, the use of GPT-based models has substantially increased since 2023.

show abstract

Reply to Moreno et al. Comment on “Sallam, M. ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns. Healthcare 2023, 11, 887”

Abstract: I would like to thank the authors for their commentary on the publication “ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns” [...]

Cited by 8 publications

References 11 publications

The Performance of OpenAI ChatGPT-4 and Google Gemini in Virology Multiple-Choice Questions: A Comparative Analysis of English and Arabic Responses

The Performance of OpenAI ChatGPT-4 and Google Gemini in Virology Multiple-Choice Questions: A Comparative Analysis of English and Arabic Responses

Domain-specific LLM Development and Evaluation – A Case-study for Prostate Cancer

Large Language Models to process, analyze, and synthesize biomedical texts – a scoping review

Contact Info

Product

Resources

About