Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning

Tiu, Ekin; Talius, Ellie; Patel, Pujan R.; Langlotz, Curtis P.; Ng, Andrew Y.; Rajpurkar, Pranav

doi:10.1038/s41551-022-00936-9

Cited by 149 publications

(60 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Section: Discussionmentioning

confidence: 99%

“…Unlike CNN-based models, large language models or multimodal models have been developed more recently. Publications using text data or multimodal data have been steadily increasing, and their maturity is improving 25,26 . Readily available CNN algorithms and large imaging data repositories enabled radiology and other image-based specialties such as ophthalmology, gastroenterology, oncology, and cardiology to generate a huge growth of mature model publication.…”

mentioning

confidence: 99%

See 1 more Smart Citation

Quantitative and Qualitative evaluation of the recent Artificial Intelligence in Healthcare publications using Deep-Learning

Awasthi

Mishra

Cywiński

et al. 2023

Preprint

View full text Add to dashboard Cite

Background: An ever increasing number of artificial intelligence (AI) models targeting healthcare applications are developed and published every day, but their use in real world decision making is limited. Beyond a quantitative assessment, it is important to have qualitative evaluation of the maturity of these publications with additional details related to trends in type of data used, type of models developed across the healthcare spectrum. Methods: We assessed the maturity of selected peer-reviewed AI publications pertinent to healthcare for 2019 to 2021. For the report, the data collection was performed by PubMed search using machine learning OR artificial intelligence AND Healthcare with the English language and human subject research as of December 31, each year. All three years selected were manually classified into 34 distinct medical specialties. We used the Bidirectional Encoder Representations from Transformers (BERT) neural networks model to identify the maturity level of research publications based on their abstracts. We further classified a mature publication based on the healthcare specialty and geographical location of the article's senior author. Finally, we manually annotated specific details from mature publications, such as model type, data type, and disease type. Results: Of the 7062 publications relevant to AI in healthcare from 2019 to 2021, 385 were classified as mature. In 2019, 6.01 percent of publications were mature. 7.7 percent were mature in 2020, and 1.81 percent of publications were mature in 2021. Radiology publications had the most mature model publications across all specialties over the last three years, followed by pathology in 2019, ophthalmology in 2020, and gastroenterology in 2021. Geographical pattern analysis revealed a non-uniform distribution pattern. In 2019 and 2020, the United States ranked first with a frequency of 22 and 50, followed by China with 20 and 47. In 2021, China ranked first with 17 mature articles, followed by the United States with 11 mature articles. Imaging based data was the primary source, and deep learning was the most frequently used modeling technique in mature publications. Interpretation: Despite the growing number of publications of AI models in healthcare, only a few publications have been found to be mature with a potentially positive impact on healthcare. Globally, there is an opportunity to leverage diverse datasets and models across the health spectrum, to develop more mature models and related publications, which can fully realize the potential of AI to transform healthcare.

show abstract

Section: Discussionmentioning

confidence: 99%

mentioning

confidence: 99%

Quantitative and Qualitative evaluation of the recent Artificial Intelligence in Healthcare publications using Deep-Learning

Awasthi

Mishra

Cywiński

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…For instance, within months after its release, GPT-3 powered more than 300 apps across various industries 42 . As a promising early example of a medical foundation model, CheXzero can be applied to detect dozens of diseases in chest X-rays without being trained on explicit labels for these diseases 9 . Likewise, the shift towards GMAI will drive the development and release of large-scale medical AI models with broad capabilities, which will form the basis for various downstream clinical applications.…”

Section: Adaptabilitymentioning

confidence: 99%

“…Although there have been early efforts to develop medical foundation models [8][9][10][11] , this shift has not yet widely permeated medical AI, owing to the difficulty of accessing large, diverse medical datasets, the complexity of the medical domain and the recency of this development. Instead, medical AI models are largely still developed with a task-specific approach to model development.…”

mentioning

confidence: 99%

Foundation models for generalist medical artificial intelligence

Moor

Banerjee

Abad

et al. 2023

Nature

Self Cite

533

125

View full text Add to dashboard Cite

The exceptionally rapid development of highly flexible, reusable artificial intelligence (AI) models is likely to usher in newfound capabilities in medicine. We propose a new paradigm for medical AI, which we refer to as generalist medical AI (GMAI). GMAI models will be capable of carrying out a diverse set of tasks using very little or no task-specific labelled data. Built through self-supervision on large, diverse datasets, GMAI will flexibly interpret different combinations of medical modalities, including data from imaging, electronic health records, laboratory results, genomics, graphs or medical text. Models will in turn produce expressive outputs such as free-text explanations, spoken recommendations or image annotations that demonstrate advanced medical reasoning abilities. Here we identify a set of high-impact potential applications for GMAI and lay out specific technical capabilities and training datasets necessary to enable them. We expect that GMAI-enabled applications will challenge current strategies for regulating and validating AI devices for medicine and will shift practices associated with the collection of large medical datasets.

show abstract

“…For example, "tumor" was converted to "An H&E image of tumor". As a natural comparison on the zero-shot task, we compared PLIP with the original CLIP model, which has been frequently used for other medical image tasks [31][32][33] and has already been trained from other medical images. Our analysis showed that PLIP consistently outperformed the baseline CLIP model and the results from predicting the majority class (or Majority) (Figure 2c).…”

Section: Plip Can Perform Zero-shot Classification On New Datamentioning

confidence: 99%

Leveraging medical Twitter to build a visual–language foundation model for pathology AI

Huang

Bianchi

Yüksekgönül

et al. 2023

Preprint

View full text Add to dashboard Cite

The lack of annotated publicly available medical images is a major barrier for innovations. At the same time, many de-identified images and much knowledge are shared by clinicians on public forums such as medical Twitter. Here we harness these crowd platforms to curate OpenPath, a large dataset of 208,414 pathology images paired with natural language descriptions. This is the largest public dataset for pathology images annotated with natural text. We demonstrate the value of this resource by developing PLIP, a multimodal AI with both image and text understanding, which is trained on OpenPath. PLIP achieves state-of-the-art zero-shot and transfer learning performances for classifying new pathology images across diverse tasks. Moreover, PLIP enables users to retrieve similar cases by either image or natural language search, greatly facilitating knowledge sharing. Our approach demonstrates that publicly shared medical information is a tremendous resource that can be harnessed to advance biomedical AI.

show abstract

Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning

Cited by 149 publications

References 42 publications

Quantitative and Qualitative evaluation of the recent Artificial Intelligence in Healthcare publications using Deep-Learning

Quantitative and Qualitative evaluation of the recent Artificial Intelligence in Healthcare publications using Deep-Learning

Foundation models for generalist medical artificial intelligence

Leveraging medical Twitter to build a visual–language foundation model for pathology AI

Contact Info

Product

Resources

About