Multimodal medical tensor fusion network-based DL framework for abnormality prediction from the radiology CXRs and clinical text reports

Abstract

Pulmonary disease is a commonly occurring abnormality throughout this world. The pulmonary diseases include Tuberculosis, Pneumothorax, Cardiomegaly, Pulmonary atelectasis, Pneumonia, etc. A timely prognosis of pulmonary disease is essential. Increasing progress in Deep Learning (DL) techniques has significantly impacted and contributed to the medical domain, specifically in leveraging medical imaging for analysis, prognosis, and therapeutic decisions for clinicians. Many contemporary DL strategies for radiology focus on a single modality of data utilizing imaging features without considering the clinical context that provides more valuable complementary information for clinically consistent prognostic decisions. Also, the selection of the best data fusion strategy is crucial when performing Machine Learning (ML) or DL operation on multimodal heterogeneous data. We investigated multimodal medical fusion strategies leveraging DL techniques to predict pulmonary abnormality from the heterogeneous radiology Chest X-Rays (CXRs) and clinical text reports. In this research, we have proposed two effective unimodal and multimodal subnetworks to predict pulmonary abnormality from the CXR and clinical reports. We have conducted a comprehensive analysis and compared the performance of unimodal and multimodal models. The proposed models were applied to standard augmented data and the synthetic data generated to check the model’s ability to predict from the new and unseen data. The proposed models were thoroughly assessed and examined against the publicly available Indiana university dataset and the data collected from the private medical hospital. The proposed multimodal models have given superior results compared to the unimodal models.

Cross-modal Deep Learning-based Clinical Recommendation System for Radiology Report Generation from Chest X-rays

“…The suggested multichannel dilation convolution layer provides more comprehensive imaging data by generating a larger receptive field, while keeping the network parameters constant, in contrast to the traditional convolutional layer. Moreover, to ensure an even distribution of computational workload across each layer, the Depthwise Separable convolution network is utilized instead of the conventional convolution network [23]. The UM-VES framework is used to extract visual features from both the frontal and lateral CXR images independently, and the resulting features are combined by concatenation.…”

“…The resulting features are then concatenated and passed through 13 depthwise separable layers to learn and extract additional features. For a more comprehensive understanding of the model, readers can refer to our previous paper [23], where we provide a detailed overview and description of the UM-VES model's architecture and components.…”

“…The detailed architecture of the proposed UM-TES model is shown in Figure 4. To gain a more thorough understanding of the UM-TES model's architecture and components, readers can refer to our previous paper [23], where we offer a detailed overview and description.…”

Diagnostic Performance Evaluation of Deep Learning-Based Medical Text Modelling to Predict Pulmonary Diseases from Unstructured Radiology Free-Text Reports

“…The unstructured nature of radiology free-text reports with complex vocabularies makes it difficult for ML/DL models to extract features from the raw text. NLP plays a key role in extracting structured information from clinical text (Pons et al, 2016;Shetty et al, 2023).…”