Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training

Moon, Jong Hak; Lee, Hyungyung; Shin, Woncheol; Kim, Young‐Hak; Choi, Edward

doi:10.48550/arxiv.2105.11333

Cited by 4 publications

(4 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, [8] proposed to perform the pre-training on medical image-text pairs to capture medical knowledge, but their evaluation was conducted only on Med-VQA despite the promising improvement is observed. The most related work to ours is [18], which pre-trained a Med-VLP model and verified its effectiveness on various downstream tasks. Yet it is limited to the chest X-ray, and more importantly, the pre-training was not performed in a self-supervised manner (i.e., using the diagnosis labels).…”

Section: Introductionmentioning

confidence: 90%

Multi-modal Masked Autoencoders for Medical Vision-and-Language Pre-training

Chen

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Medical vision-and-language pre-training provides a feasible solution to extract effective vision-and-language representations from medical images and texts. However, few studies have been dedicated to this field to facilitate medical vision-and-language understanding. In this paper, we propose a self-supervised learning paradigm with multimodal masked autoencoders (M 3 AE), which learn cross-modal domain knowledge by reconstructing missing pixels and tokens from randomly masked images and texts. There are three key designs to make this simple approach work. First, considering the different information densities of vision and language, we adopt different masking ratios for the input image and text, where a considerably larger masking ratio is used for images. Second, we use visual and textual features from different layers to perform the reconstruction to deal with different levels of abstraction in visual and language. Third, we develop different designs for vision and language decoders (i.e., a Transformer for vision and a multi-layer perceptron for language). To perform a comprehensive evaluation and facilitate further research, we construct a medical vision-and-language benchmark including three tasks. Experimental results demonstrate the effectiveness of our approach, where state-of-the-art results are achieved on all downstream tasks. Besides, we conduct further analysis to better verify the effectiveness of different components of our approach and various settings of pre-training. The source code is available at https: //github.com/zhjohnchan/M3AE.

show abstract

Section: Introductionmentioning

confidence: 90%

Multi-modal Masked Autoencoders for Medical Vision-and-Language Pre-training

Chen

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…Another example would be language translation where GLLMMs can convert text from one language to another with greater accuracy because they have also been primed by gaining earlier access to cultural nuances and context [12]. An example where GLLMMs are applied in the field of medicine would be to analyze the results of medical imaging more accurately (such as X-ray, MRI scans, or ultrasound) by pretraining with access to additional databases that relate to the relevant histology and pathology implicated in the imaging results [13].…”

Section: A Brief Primer On the Concepts And Nomenclature Of Aimentioning

confidence: 99%

Artificial Intelligence and Healthcare Simulation: The Shifting Landscape of Medical Education

Hamilton

2024

Cureus

View full text Add to dashboard Cite

The impact of artificial intelligence (AI) will be felt not only in the arena of patient care and deliverable therapies but will also be uniquely disruptive in medical education and healthcare simulation (HCS), in particular. As HCS is intertwined with computer technology, it offers opportunities for rapid scalability with AI and, therefore, will be the most practical place to test new AI applications. This will ensure the acquisition of AI literacy for graduates from the country's various healthcare professional schools. Artificial intelligence has proven to be a useful adjunct in developing interprofessional education and team and leadership skills assessments. Outcome-driven medical simulation has been extensively used to train students in imagecentric disciplines such as radiology, ultrasound, echocardiography, and pathology. Allowing students and trainees in healthcare to first apply diagnostic decision support systems (DDSS) under simulated conditions leads to improved diagnostic accuracy, enhanced communication with patients, safer triage decisions, and improved outcomes from rapid response teams. However, the issue of bias, hallucinations, and the uncertainty of emergent properties may undermine the faith of healthcare professionals as they see AI systems deployed in the clinical setting and participating in diagnostic judgments. Also, the demands of ensuring AI literacy in our healthcare professional curricula will place burdens on simulation assets and faculty to adapt to a rapidly changing technological landscape. Nevertheless, the introduction of AI will place increased emphasis on virtual reality platforms, thereby improving the availability of self-directed learning and making it available 24/7, along with uniquely personalized evaluations and customized coaching. Yet, caution must be exercised concerning AI, especially as society's earlier, delayed, and muted responses to the inherent dangers of social media raise serious questions about whether the American government and its citizenry can anticipate the security and privacy guardrails that need to be in place to protect our healthcare practitioners, medical students, and patients.

show abstract

“…Recent research utilizing large data via pre-training Transformer for medical applications are mainly related to visionand-language (VL) learning. Moon et al [14] pre-trains a Transformer-based model on aligned X-ray images and associated reports for learning joint VL representations in the medical domain. The downstream tasks include both comprehension and generation tasks.…”

Section: Related Workmentioning

confidence: 99%

A Self-Supervised Pre-Training Framework for Vision-Based Seizure Classification

Hou

McGonigal

Bartolomei

et al. 2022

ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

View full text Add to dashboard Cite

Seizure events feature temporary abnormalities in muscle control or movements. They are usually caused by excessive neuronal activities in the brain, and are called epileptic seizures (ES). Nevertheless, not all seizures are epileptic in origin. Some are caused by psychological reasons, and such type of seizures are called psychogenic non-epileptic seizures (PNES). We propose a method to classify ES and PNES based on clinical signs in the seizure videos. In particular, inspired by BERT, we propose a Transformer-based framework that pre-trains on large unlabeled clinical videos, and then we fine-tune the pre-trained model for seizure classification with a minimum modification. We conduct a leave-one-subjectout (LOSO) validation on our dataset. The F1-score and accuracy are 0.82 and 0.75, respectively. To our knowledge, the proposed approach is the first attempt to use large unannotated data and learn useful representations for downstream tasks in the field of video based seizure analysis.

show abstract

Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training

Cited by 4 publications

References 35 publications

Multi-modal Masked Autoencoders for Medical Vision-and-Language Pre-training

Multi-modal Masked Autoencoders for Medical Vision-and-Language Pre-training

Artificial Intelligence and Healthcare Simulation: The Shifting Landscape of Medical Education

A Self-Supervised Pre-Training Framework for Vision-Based Seizure Classification

Contact Info

Product

Resources

About