Advancing Radiograph Representation Learning with Masked Record Modeling

Zhou, Haoming; Lian, Chenyu; Wang, Liansheng; Yu, Yizhou

doi:10.48550/arxiv.2301.13155

Cited by 3 publications

(4 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In recent years, contrastive learning, which trains the models to distinguish between positive and negative pairs, has achieved state-of-the-art performances in visual representation learning (Chen et al, 2020a;He et al, 2020;Chen et al, 2020b) and vision-language representation learning (Radford et al, 2021). Inspired by the great success of contrastive learning, several works (Huang et al, 2021;Zhang et al, 2020a;Boecking et al, 2022;Zhou et al, 2023Wang et al, 2022b) have been proposed to learn robust and accurate medical vision-language representations, which can be used to achieve promising results on various downstream tasks. However, most existing works focus on contrasting images with entire reports, ignoring the potential phenotypes in each sentence within the reports.…”

Section: Contrastive Learningmentioning

confidence: 99%

See 1 more Smart Citation

Fine-grained Medical Vision-Language Representation Learning for Radiology Report Generation

Wang,

Peng,

Liu

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

Given the input radiology images, the objective of medical report generation is to produce accurate and comprehensive medical reports, which typically include multiple descriptive clinical sentences associated with different phenotypes. Most existing approaches have relied on a pretrained vision encoder to extract the visual representations of the images. In this study, we propose a phenotype-based contrastive learning framework, i.e., PhenotypeCLIP, to efficiently bridge the gap between visual and textual modalities for improved text generation. In contrast to existing contrastive learning methods which learn representations by contrasting images with entire reports, our approach learns more fine-grained representations, i.e., phenotype-based representations, by contrasting images with each sentence within the reports. The experiments on two widely-used datasets MIMIC-CXR and IU X-ray demonstrate that PhenotypeCLIP can achieve promising performances and substantially outperform the conventional contrastive learning methods.

show abstract

Section: Contrastive Learningmentioning

confidence: 99%

“…As we can see, although contrastive learning has been well explored for medical representation learning in existing literature (Huang et al, 2021;Zhang et al, 2020a;Boecking et al, 2022;Zhou et al, 2023Wang et al, 2022b), they encode the entire report y i for training.…”

Section: Phenotypeclipmentioning

confidence: 99%

Fine-grained Medical Vision-Language Representation Learning for Radiology Report Generation

Wang,

Peng,

Liu

et al. 2023

Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

View full text Add to dashboard Cite

show abstract

“…However, collecting labelled data for a rare disease is expensive and time-consuming. To this end, as shown in Figure 1, we propose the Unsupervised Learning from Unlabelled Medical Images and Text (ULUMIT) framework for radiograph representation learning 26,27,28,29 , which deals with the situation where the labelled data are scarce, to shorten this time-frame, allowing us to respond quickly in future to rare diseases.…”

Section: Introductionmentioning

confidence: 99%

A Multimodal Large Language Modelling Deep Learning Framework for the Future Pandemic

Liu,

Zhu,

et al. 2023

Preprint

View full text Add to dashboard Cite

Deep neural networks have been integrated into the whole clinical decision procedure which can improve the efficiency of diagnosis and alleviate the heavy workload of physicians. Typical applications include 1) medical report generation, 2) disease classification, and 3) survival prediction. Since most neural networks are supervised, their quality heavily depends on the volume and quality of available labels. However, for rare diseases, e.g., new pandemics, there are few existing labels. In addition, collecting sufficient labels for training is time-consuming and is typically unavailable at the early stage. In this paper, we propose a multimodal large language model - Unsupervised Learning from Unlabelled Medical Images and Text (ULUMIT) framework for radiograph representation learning, which can learn broad medical knowledge (e.g., image understanding, text semantics, and clinical phenotypes) from unlabelled data. As a result, when encountering a rare disease, our framework can be rapidly deployed and easily adapted to them with limited labels. Furthermore, ULUMIT supports medical data across visual modality (e.g., chest X-ray and CT) and textual modality (e.g., medical report and free-text clinical note); therefore, it can be used for clinical tasks that involve both visual and textual medical data. We demonstrate the effectiveness of our ULUMIT by showing how it would perform using the COVID-19 pandemic ``in replay''. In particular, in the retrospective setting, we test the model on the early COVID-19 datasets; and in the prospective setting, we test the model on the new variant COVID-19-Omicron. The experiments are conducted on 1) three kinds of input medical data, image-only, text-only, and image-text; 2) three kinds of downstream tasks, medical reporting, diagnosis, and prognosis; 3) five public COVID-19 datasets; and 4) three different languages, i.e., English, Chinese, and Spanish. All experiments consistently show that our framework can make accurate and robust COVID-19 decision-support tasks with little labelled data. Besides COVID-19, our framework can be applied to identify 14 common thorax diseases and tuberculosis across five additional public datasets, demonstrating its robustness in generalization and transferability. In brief, our framework achieves state-of-the-art performances on ten datasets.

show abstract

“…439mark datasets for common disease classification tasks. We 440 follow previous works64,58,59,72,66 to pre-process the datasets 441 and perform the evaluation. As we can see fromTable 5, 442 with limited labels (1%), our method can achieve competi-443 tive results with previous fully-supervised methods trained 444 on full labels across all evaluation datasets.…”

mentioning

confidence: 99%

A Large Language Modelling Deep Learning Framework for the Next Pandemic

Zhu

Wang

Yang

et al. 2023

Preprint

View full text Add to dashboard Cite

Deep neural networks have been integrated into the whole clinical decision procedure which can improve the efficiency of diagnosis and alleviate the heavy workload of physicians. Typical applications include 1) medical report generation, 2) disease classification, and 3) survival prediction. Since most neural networks are supervised, their quality heavily depends on the volume and quality of available labels. However, for novel diseases, e.g., new pandemics or new variants, there are few existing labels. In addition, the acquisition of new pandemic cases to collect sufficient labels for training is time-consuming and is typically unavailable at the early stage. To prepare neural networks for the next pandemic, in this paper, we propose a large language model - Unsupervised Learning from Unlabelled Medical Images and Text (ULUMIT) framework, which can learn broad medical knowledge (e.g., image understanding, text semantics, and clinical phenotypes) from unlabelled data. As a result, when encountering new pandemics, our framework can be rapidly deployed and easily adapted to them with extremely limited labels. Furthermore, ULUMIT supports medical data across visual modality (e.g., chest X-ray and CT) and textual modality (e.g., medical report and free-text clinical note); therefore, it can be used for any clinical task that involves both visual and textual medical data. We demonstrate the effectiveness of our ULUMIT by showing how it would perform using the COVID-19 pandemic ``in replay''. In particular, in the retrospective setting, we test the model on the early COVID-19 datasets; and in the prospective setting, we test the model on the new variant COVID-19-Omicron. The experiments are conducted on 1) three kinds of input medical data, image-only, text-only, and image-text; 2) three kinds of downstream tasks, medical reporting, diagnosis, and prognosis; 3) five public COVID-19 datasets; and 4) three different languages, i.e., English, Chinese, and Spanish. All experiments consistently show that our framework can make accurate and robust COVID-19 decision-support tasks with little labelled data (such as considering information from only one patient), providing an impact on medical data analysis during the early stage of the next pandemic. Besides COVID-19, our framework can be applied to identify 14 common thorax diseases and tuberculosis across five additional public datasets, demonstrating its robustness in generalization and transferability. In brief, our framework achieves state-of-the-art performances on ten datasets.

show abstract

Advancing Radiograph Representation Learning with Masked Record Modeling

Cited by 3 publications

References 27 publications

Fine-grained Medical Vision-Language Representation Learning for Radiology Report Generation

Fine-grained Medical Vision-Language Representation Learning for Radiology Report Generation

A Multimodal Large Language Modelling Deep Learning Framework for the Future Pandemic

A Large Language Modelling Deep Learning Framework for the Next Pandemic

Contact Info

Product

Resources

About