On the Opportunities and Risks of Foundation Models for Natural Language Processing in Radiology

Wiggins, Walter F.; Tejani, Ali S.

doi:10.1148/ryai.220119

Cited by 27 publications

(12 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…NLP has received considerable media attention in the past months due to the release of large DL models called foundation models 5 . These models can be repurposed for various tasks, such as generating text or images, after being trained on a wide range of unlabelled data 6 7 . A prominent example of a foundation model is Generative Pre-trained Transformer 3 (GPT-3), a large language model (LLM) that generates human-like text.…”

Section: Introductionmentioning

confidence: 99%

Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of its Successes and Shortcomings

Antaki

Touma

Milad

et al. 2023

Preprint

150

121

View full text Add to dashboard Cite

We tested the accuracy of ChatGPT, a large language model (LLM), in the ophthalmology question-answering space using two popular multiple choice question banks used for the high-stakes Ophthalmic Knowledge Assessment Program (OKAP) exam. The testing sets were of easy-to-moderate difficulty and were diversified, including recall, interpretation, practical and clinical decision-making problems. ChatGPT achieved 55.8% and 42.7% accuracy in the two 260-question simulated exams. Its performance varied across subspecialties, with the best results in general medicine and the worst in neuro-ophthalmology and ophthalmic pathology and intraocular tumors. These results are encouraging but suggest that specialising LLMs through domain-specific pre-training may be necessary to improve their performance in ophthalmic subspecialties.

show abstract

Section: Introductionmentioning

confidence: 99%

Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of its Successes and Shortcomings

Antaki

Touma

Milad

et al. 2023

Preprint

150

121

View full text Add to dashboard Cite

show abstract

“… Foundation AI models were initially introduced in natural language processing in 2018, but have since been applied in computer vision and robotics. 10 – 12 Visual transformers revolutionized how AI models approach sequential data and images by allowing complex predictions in smaller datasets. They are enabled by transfer learning and scale.…”

Section: Reviewmentioning

confidence: 99%

“…6 –9 While improving model performance was limited over the past decade to optimizing architectures and scaling up the training datasets, both strategies are particularly challenging in epilepsy surgery where cohorts are typically small. Foundation AI models were initially introduced in natural language processing in 2018, but have since been applied in computer vision and robotics. 10 –12 Visual transformers revolutionized how AI models approach sequential data and images by allowing complex predictions in smaller datasets. They are enabled by transfer learning and scale.…”

Section: Reviewmentioning

confidence: 99%

Machine Learning for Precision Epilepsy Surgery

Jehi

2023

Epilepsy Curr

View full text Add to dashboard Cite

Brain surgery offers the best chance of seizure-freedom for patients with focal drug-resistant epilepsy, but only 50% achieve sustained seizure-freedom. With the explosion of data collected during routine presurgical evaluations and recent advances in computational science, we now have a tremendous potential to achieve precision epilepsy surgery: a data-driven tailoring of surgical planning. This review highlights the clinical need, the relevant computational science focusing on machine learning, and discusses some specific applications in epilepsy surgery.

show abstract

“…Foundation models have presented impressive zero-shot and few-shot generalizations [22], [23]. Such progresses have been extended for training multi-modal fundamental models like CLIP [24].…”

Section: Segment Anything Modelmentioning

confidence: 99%

FakeLocator: Robust Localization of GAN-Based Face Manipulations

Huang

Juefei-Xu

Guo

et al. 2022

IEEE Trans.Inform.Forensic Secur.

View full text Add to dashboard Cite

Segment anything model (SAM) has presented impressive objectness identification capability with the idea of prompt learning and a new collected large-scale dataset. Given a prompt (e.g., points, bounding boxes, or masks) and an input image, SAM is able to generate valid segment masks for all objects indicated by the prompts, presenting high generalization across diverse scenarios and being a general method for zero-shot transfer to downstream vision tasks. Nevertheless, it remains unclear whether SAM may introduce errors in certain threatening scenarios. Clarifying this is of significant importance for applications that require robustness, such as autonomous vehicles. In this paper, we aim to study the testing-time robustness of SAM under adversarial scenarios and common corruptions. To this end, we first build a testing-time robustness evaluation benchmark for SAM by integrating existing public datasets. Second, we extend representative adversarial attacks against SAM and study the influence of different prompts on robustness. Third, we study the robustness of SAM under diverse corruption types by evaluating SAM on corrupted datasets with different prompts. With experiments conducted on SA-1B and KITTI datasets, we find that SAM exhibits remarkable robustness against various corruptions, except for blur-related corruption. Furthermore, SAM remains susceptible to adversarial attacks, particularly when subjected to PGD and BIM attacks. We think such a comprehensive study could highlight the importance of the robustness issues of SAM and trigger a series of new tasks for SAM as well as downstream vision tasks.

show abstract

On the Opportunities and Risks of Foundation Models for Natural Language Processing in Radiology

Cited by 27 publications

References 4 publications

Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of its Successes and Shortcomings

Evaluating the Performance of ChatGPT in Ophthalmology: An Analysis of its Successes and Shortcomings

Machine Learning for Precision Epilepsy Surgery

FakeLocator: Robust Localization of GAN-Based Face Manipulations

Contact Info

Product

Resources

About