MedKLIP: Medical Knowledge Enhanced Language-Image Pre-Training

Abstract

In this paper, we consider the problem of enhancing self-supervised visual-language pre-training (VLP) with medical-specific knowledge, by exploiting the paired image-text reports from the radiological daily practice. In particular, we make the following contributions: First, unlike existing works that directly process the raw reports, we adopt a novel report filter to extract the medical entities, avoiding unnecessary complexity from language grammar and enhancing the supervision signals; Second, we propose a novel entity embedding module by querying an external knowledge description base, to exploit the rich context of additional information that the medical domain affords, and implicitly build relationships between entities in the language embedding space; Third, we propose a novel Transformer-based fusion model for spatially aligning the entity description with visual signals at the image patch level only with self-supervised learning, thus enabling the ability for spatial grounding; Fourth, we conduct thorough experiments to validate the effectiveness of our proposed architecture, and benchmark on numerous public benchmarks e.g., ChestX-ray14, RSNA Pneumonia, SIIM-ACR Pneumothorax, COVIDx CXR-2, COVID Rural, and EdemaSeverity. In both zero-shot and fine-tuning settings, our model has demonstrated strong performance compared with the former methods on disease classification and grounding.

Advancing Radiograph Representation Learning with Masked Record Modeling

“…Moreover, REFERS introduced a multi-view fusion attention to better align the representations of each radiograph and its associated report. In addition, MGCA and Med-KLIP (Wu et al, 2023) were included as two recent baselines 4 . Apart from above baselines, we also include M3AE (Geng et al, 2022), a recent masked multi-modal pre-training method aside from the application to medical data, for comparison.…”

“…MGCA and MedKLIP(Wu et al, 2023) were released after the submission deadline of ICLR 2023. We added results in the camera ready for better comparisons.…”

Improvement of Outpatient Service Processes: A Case Study of the University of Hong Kong-Shenzhen Hospital

“…In general, most existing Med-VLP models can be classified into two types: the dual-encoder type and the fusion-encoder type, where the former encodes images and texts separately to learn uni-modal/cross-modal representations following a shallow interaction layer (i.e., an image-text contrastive layer), and the latter performs an early fusion of the two modalities through the selfattention/co-attention mechanisms to learn multi-modal representations. 2 For dual-encoders, the purpose of existing studies [66,19,44,60,57,61,3] is to develop label-efficient algorithms to learn effective uni-modal/cross-modal representations since large-scale manually labeled datasets are difficult and expensive to obtain for medical images. The learned representations can improve the effectiveness of uni-modal (i.e., vision-only or language-only) tasks 3 and the efficiency of crossmodal (i.e., image-to-text or text-to-image) retrieval tasks significantly.…”

MGA: Medical generalist agent through text-guided knowledge transformation

“…However, existing methods are still limited in the following aspects: 1) Model redundancy caused by multiple branches or multiple downstream heads. They tend to treat the unified model as a feature extractor to extract common features, and add different network branches for the specific tasks [7,13,17,[21][22][23]. For example, there are works using a large language model (i.e.…”