Artificial Intelligence Training in Canadian Dermatology to Increase Dermatologists Engagement and Enhance Medical Practice

Joly-Chevrier, Maxine; Lefrançois, Pascale

doi:10.1177/12034754221143081

Cited by 4 publications

(4 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[30] While LLMs such as ChatGPT undoubtedly possess substantial potential in several domains it has demonstrated significant limitations in medicine and healthcare. [31][32][33][34] Impressive performance in one task does not necessarily translate to equivocal performance in others. Users of this technology, both medical practitioners and patients alike, need be aware.…”

Section: Discussionmentioning

confidence: 99%

Exploring the Capabilities of ChatGPT in Women’s Health

Bachmann,

Duta,

Mazey

et al. 2024

Preprint

View full text Add to dashboard Cite

Introduction: Artificial Intelligence (AI) is redefining healthcare, with Large Language Models (LLMs) like ChatGPT offering novel and powerful capabilities in processing and generating human-like information. These advancements offer potential improvements in Women's Health, particularly Obstetrics and Gynaecology (O&G), where diagnostic and treatment gaps have long existed. Despite its generalist nature, ChatGPT is increasingly being tested in healthcare, necessitating a critical analysis of its utility, limitations and safety. This study examines ChatGPT's performance in interpreting and responding to international gold standard benchmark assessments in O&G: the RCOG's MRCOG Part One and Two examinations. We evaluate ChatGPT's domain- and knowledge area-specific accuracy, the influence of linguistic complexity on performance and its self-assessment confidence and uncertainty, essential for safe clinical decision-making. Methods: A dataset of MRCOG examination questions from sources beyond the reach of LLMs was developed to mitigate the risk of ChatGPT's prior exposure. A dual-review process validated the technical and clinical accuracy of the questions, omitting those dependent on previous content, duplicates, or requiring image interpretation. Single Best Answer (SBA) and Extended Matching (EMQ) Questions were converted to JSON format to facilitate ChatGPT's interpretation, incorporating question types and background information. Interaction with ChatGPT was conducted via OpenAI's API, structured to ensure consistent, contextually informed responses from ChatGPT. The response from ChatGPT was recorded and compared against the known accurate response. Linguistic complexity was evaluated using unique token counts and Type-Token ratios (vocabulary breadth and diversity) to explore their influence on performance. ChatGPT was instructed to assign confidence scores to its answers (0-100%), reflecting its self-perceived accuracy. Responses were categorized by correctness and statistically analysed through entropy calculation, assessing ChatGPT's capacity for self-evaluating certainty and knowledge boundaries. Findings: Of 1,824 MRCOG Part One and Two questions, ChatGPT's accuracy on MRCOG Part One was 72.2% (95% CI 69.2-75.3). For Part Two, it achieved 50.4% accuracy (95% CI 47.2-53.5) with 534 correct out of 989 questions, performing better on SBAs (54.0%, 95% CI 50.0-58.0) than on EMQs (45.0%, 95% CI 40.1-49.9). In domain-specific performance, the highest accuracy was in Biochemistry (79.8%, 95% CI 71.4-88.1) and the lowest in Biophysics (51.4%, 95% CI 35.2-67.5). The best-performing subject in Part Two was Urogynaecology (63.0%, 95% CI 50.1-75.8) and the worst was Management of Labour (35.6%, 95% CI 21.6-49.5). Linguistic complexity analysis showed a marginal increase in unique token count for correct answers in Part One (median 122, IQR 114-134) compared to incorrect (median 120, IQR 112-131, p=0.05). TTR analysis revealed higher medians for correct answers with negligible effect sizes (Part One: 0.66, IQR 0.63-0.68; Part Two: 0.62, IQR 0.57-0.67) and p-values <0.001. Regarding self-assessed confidence, the median confidence for correct answers was 70.0% (IQR 60-90), the same as for incorrect choices identified as correct (p<0.001). For correct answers deemed incorrect, the median confidence was 10.0% (IQR 0-10), and for incorrect answers accurately identified, it was 5.0% (IQR 0-10, p<0.001). Entropy values were identical for correct and incorrect responses (median 1.46, IQR 0.44-1.77), indicating no discernible distinction in ChatGPT's prediction certainty. Conclusions: ChatGPT demonstrated commendable accuracy in basic medical queries on the MRCOG Part One, yet its performance was markedly reduced in the clinically demanding Part Two exam. The model's high self-confidence across correct and incorrect responses necessitates scrutiny for its application in clinical decision-making. These findings suggest that while ChatGPT has potential, its current form requires significant refinement before it can enhance diagnostic efficacy and clinical workflow in women's health.

show abstract

Section: Discussionmentioning

confidence: 99%

Exploring the Capabilities of ChatGPT in Women’s Health

Bachmann,

Duta,

Mazey

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…As technologies relying on DL algorithms will become more prevalent in dermatology practice, digital health training would be pertinent to inform healthcare professionals of the underlying mechanisms behind these technologies. 46 …”

Section: Discussionmentioning

confidence: 99%

The State of Artificial Intelligence in Skin Cancer Publications

Joly-Chevrier,

Nguyen,

Liang

et al. 2024

J Cutan Med Surg

Self Cite

View full text Add to dashboard Cite

Background: Artificial intelligence (AI) in skin cancer is a promising research field to assist physicians and to provide support to patients remotely. Physicians’ awareness to new developments in AI research is important to define the best practices and scope of integrating AI-enabled technologies within a clinical setting. Objectives: To analyze the characteristics and trends of AI skin cancer publications from dermatology journals. Methods: AI skin cancer publications were retrieved in June 2022 from the Web of Science. Publications were screened by title, abstract, and keywords to assess eligibility. Publications were fully reviewed. Publications were divided between nonmelanoma skin cancer (NMSC), melanoma, and skin cancer studies. The primary measured outcome was the number of citations. The secondary measured outcomes were articles’ general characteristics and features related to AI. Results: A total of 168 articles were included: 25 on NMSC, 77 on melanoma, and 66 on skin cancer. The most common types of skin cancers were melanoma (134, 79.8%), basal cell carcinoma (61, 36.3%), and squamous cell carcinoma (45, 26.9%). All articles were published between 2000 and 2022, with 49 (29.2%) of them being published in 2021. Original studies that developed or assessed an algorithm predominantly used supervised learning (66, 97.0%) and deep neural networks (42, 67.7%). The most used imaging modalities were standard dermoscopy (76, 45.2%) and clinical images (39, 23.2%). Conclusions: Most publications focused on developing or assessing screening technologies with mainly deep neural network algorithms. This indicates the eminent need for dermatologists to label or annotate images used by novel AI systems.

show abstract

“…They allow researchers to quickly grasp the key aspects of a topic without having to sift through numerous individual studies. The increasing popularity of ChatGPT among healthcare professionals and researchers alike leads to a surge in research focusing on the evaluation of ChatGPT’s performance in different application scenarios, such as medical consultation [1], research [2], education [3], or different medical specialties, such as neurology [4, 5], pediatric [6, 7], cosmetic surgery [8, 9] and dermatology [10, 11]. Each of these fields presents unique challenges and opportunities for ChatGPT.…”

Section: Introductionmentioning

confidence: 99%

Medical ChatGPT – A systematic Meta-Review

Egger,

Sallam,

Luijten

et al. 2024

Preprint

View full text Add to dashboard Cite

Since its release at the end of 2022, ChatGPT has seen a tremendous rise in attention, not only from the general public, but also from medical researchers and healthcare professionals. ChatGPT definitely changed the way we can communicate now with computers. We still remember the limitations of (voice) assistants, like Alexa or Siri, that were "overwhelmed" by a follow-up question after asking about the weather, not to mention even more complex questions, which they could not handle at all. ChatGPT and other Large Language Models (LLMs) turned that in the meantime upside down. They allow fluent and continuous conversations on a human-like level with very complex sentences and diffused in the meantime into all kinds of applications and areas. One area that was not spared from this development, is the medical domain. An indicator for this is the medical search engine PubMed, which comprises currently more than 36 million citations for biomedical literature from MEDLINE, life science journals, and online books. As of March 2024, the search term "ChatGPT" already returns over 2,700 results. In general, it takes some time, until reviews, and especially systematic reviews appear for a "new" topic or discovery. However, not for ChatGPT, and the additional search restriction to "systematic review" for article type under PubMed, returns still 31 contributions, as of March 19 2024. After filtering out non-systematic reviews from the returned results, 19 publications are included. In this meta-review, we want to take a closer look at these contributions on a higher level and explore the current evidence of ChatGPT in the medical domain, because systematic reviews belong to the highest form of knowledge in science.

show abstract

Artificial Intelligence Training in Canadian Dermatology to Increase Dermatologists Engagement and Enhance Medical Practice

Cited by 4 publications

References 4 publications

Exploring the Capabilities of ChatGPT in Women’s Health

Exploring the Capabilities of ChatGPT in Women’s Health

The State of Artificial Intelligence in Skin Cancer Publications

Medical ChatGPT – A systematic Meta-Review

Contact Info

Product

Resources

About