Pay attention to doctor–patient dialogues: Multi-modal knowledge graph attention image-text embedding for COVID-19 diagnosis

Abstract

The sudden increase in coronavirus disease 2019 (COVID-19) cases puts high pressure on healthcare services worldwide. At this stage, fast, accurate, and early clinical assessment of the disease severity is vital. In general, there are two issues to overcome: (1) Current deep learning-based works suffer from multimodal data adequacy issues; (2) In this scenario, multimodal (e.g., text, image) information should be taken into account together to make accurate inferences. To address these challenges, we propose a multi-modal knowledge graph attention embedding for COVID-19 diagnosis. Our method not only learns the relational embedding from nodes in a constituted knowledge graph but also has access to medical knowledge, aiming at improving the performance of the classifier through the mechanism of medical knowledge attention. The experimental results show that our approach significantly improves classification performance compared to other state-of-the-art techniques and possesses robustness for each modality from multi-modal data. Moreover, we construct a new COVID-19 multi-modal dataset based on text mining, consisting of 1393 doctor-patient dialogues and their 3706 images (347 X-ray 2598 CT 761 ultrasound) about COVID-19 patients and 607 non-COVID-19 patient dialogues and their 10754 images (9658 X-ray 494 CT 761 ultrasound), and the fine-grained labels of all. We hope this work can provide insights to the researchers working in this area to shift the attention from only medical images to the doctor-patient dialogue and its corresponding medical images.

Knowledge Graph Applications in Medical Imaging Analysis: A Scoping Review

“…Since 2020, knowledge graphs have also been explored in COVID-19–related research and shown noticeable performance improvements. Zheng et al pointed out that current deep learning methods suffered from data adequacy issues and that multimodal information should be considered together to make accurate inferences [ 51 ]. To solve this, they proposed a multimodal graph attention embedding mechanism to assist diagnosing COVID-19.…”

“…To solve this, they proposed a multimodal graph attention embedding mechanism to assist diagnosing COVID-19. Their method learned the relational embeddings in a constituted knowledge graph and, at the same time, improved the classifier through the medical knowledge attention mechanism [ 51 ]. According to Mudiyanselage et al, the poor performance for unseen data in COVID-19 classification can result from the limited correlation between the pretrained model and a specific imaging domain (e.g., X-ray) and the possibility of overfitting [ 52 ].…”

“…To summarize, the aforementioned work utilized different types of knowledge graphs, and they incorporated the knowledge graphs with the disease classification task using three approaches: (1) embed visual features to preconstructed prior knowledge abnormality graph [ 44 - 50 ], region graph [ 40 , 43 ], pathology graph [ 50 ], and population graph [ 42 ] and (2) extract and use visual features as graph nodes [ 41 , 52 ]; (3) use images and/or text descriptions of diagnose as graph nodes [ 49 , 51 ]. Though these work applied knowledge graphs in various ways, the results showed that incorporating knowledge graphs with disease classification achieved noticeable classification performance boosts; for example, Zhang et al achieved 1.4% improvement on average AUC, 4.7% AUC improvement on cardiomegaly, and 4.5% AUC improvement on atelectasis after adding knowledge graphs to the baseline DenseNet [ 39 ] model [ 45 ].…”

Review of Machine Learning in Lung Ultrasound in COVID-19 Pandemic

“…Combining both modalities (LUS and clinical information), they achieved an accuracy of 0.75 for four-level severity scoring and 0.88 for the binary severe/non-severe classification. On the other hand, Zheng et al built multimodal knowledge graphs from fused CT, X-ray, ultrasound, and text modalities, reaching a classification accuracy of 0.98 [ 99 ]. A multimodal channel and receptive field attention network combined with ResNeXt was proposed to process multicenter and multimodal data and achieved 0.94 accuracy [ 100 ].…”

Leveraging Representation Learning for the Construction and Application of a Knowledge Graph for Traditional Chinese Medicine: Framework Development Study

“…The model allowed the user to gain a better understanding of the drug properties from a drug similarity perspective and insights that were not easily observed in individual drugs. Zheng et al [ 32 ] took advantage of 4 kinds of modality data (X-ray images, computed tomography [CT] images, ultrasound images, and text descriptions of diagnoses) to construct a KG. The model leveraged multimodal KG attention embedding for diagnosis of COVID-19.…”