Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review

Cui, Can; Yang, Hyun; Wang, Yaohong; Zhao, Shilin; Asad, Zuhayr; Coburn, Lori A.; Wilson, Keith T.; Landman, Bennett A.; Huo, Yuankai

doi:10.1088/2516-1091/acc2fe

Cited by 49 publications

(8 citation statements)

References 109 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…By fusing multiple degraded images, DCNN-based methods can effectively de-noise and enhance the image quality, enabling better visualization and interpretation of medical conditions. These applications demonstrate the versatility and effectiveness of DCNN-based medical image fusion techniques in improving image quality, accuracy, and clinical decision-making in various healthcare scenarios [14].…”

Section: Introductionmentioning

confidence: 77%

An Ensemble Learning Approach for Multi-Modal Medical Image Fusion using Deep Convolutional Neural Networks

Godla,

Kavitha,

Ashok

et al. 2023

IJACSA

View full text Add to dashboard Cite

Medical image fusion plays a vital role in enhancing the quality and accuracy of diagnostic procedures by integrating complementary information from multiple imaging modalities. In this study, we propose an ensemble learning approach for multi-modal medical image fusion utilizing deep convolutional neural networks (DCNNs) to predict brain tumour. The proposed method aims to exploit the inherent characteristics of different modalities and leverage the power of CNNs for improved fusion results. The Generative Adversarial Network (GAN) strengthens the input images. The ensemble learning framework comprises two main stages. Firstly, a set of DCNN models is trained independently on the respective input modalities, extracting high-level features that capture modalityspecific information. Each DCNN model is fine-tuned to optimize its performance for fusion. Secondly, a fusion module is designed to aggregate the individual modality features and generate a fused image. The fusion module employs a weighted averaging technique to assign appropriate weights to the features based on their relevance and significance. The fused image obtained through this process exhibits enhanced spatial details and improved overall quality compared to the individual modalities. On a diversified dataset made up of multi-modal medical images, thorough tests are carried out to assess the efficacy of the suggested approach. The fusion images exhibit improved visual quality, enhanced feature representation, and better preservation of diagnostic information. The BRATS 2018 dataset, which contains Multi-Modal MRI images and patients' healthcare information were used. The proposed method also demonstrates robustness across different medical imaging modalities, highlighting its versatility and potential for widespread adoption in clinical practice.

show abstract

Section: Introductionmentioning

confidence: 77%

An Ensemble Learning Approach for Multi-Modal Medical Image Fusion using Deep Convolutional Neural Networks

Godla,

Kavitha,

Ashok

et al. 2023

IJACSA

View full text Add to dashboard Cite

show abstract

“…OpenAI has released plugins for images and will certainly develop multimodal foundational models in the future (e.g., GPT-4 will have capabilities to process images in the near future). But fairness research indicates that the combination with additional information or modality may not necessarily improve performance and is likely to bring about new unfairness and bias (65). For example, in CLIP, a language-vision model, historical race and gender bias are reinforced (66).…”

Section: One Lie Leads To Another: Bias Perpetuationmentioning

confidence: 99%

Surviving ChatGPT in healthcare

Liu,

Zhang,

et al. 2024

Front. Radiol.

View full text Add to dashboard Cite

At the dawn of of Artificial General Intelligence (AGI), the emergence of large language models such as ChatGPT show promise in revolutionizing healthcare by improving patient care, expanding medical access, and optimizing clinical processes. However, their integration into healthcare systems requires careful consideration of potential risks, such as inaccurate medical advice, patient privacy violations, the creation of falsified documents or images, overreliance on AGI in medical education, and the perpetuation of biases. It is crucial to implement proper oversight and regulation to address these risks, ensuring the safe and effective incorporation of AGI technologies into healthcare systems. By acknowledging and mitigating these challenges, AGI can be harnessed to enhance patient care, medical knowledge, and healthcare processes, ultimately benefiting society as a whole.

show abstract

“…In the current big data era [20] (see review for public neuroimaging datasets [19,100]), how to fully utilize multimodal information is also a significant challenge. Multimodal analysis techniques based on deep learning have made great progress [101], in which techniques such as reinforcement learning [102] and multimodal generalized foundation models [103,104] can be combined with brain parcellations to achieve better performance via the fusion of complementation information across multimodal. Importantly, it is expected to explore the interaction between microscopic phenomena, brain functions and the external environment [99], e.g.…”

Section: Hierarchy Dynamic and Multimodalitymentioning

confidence: 99%

Application of deep learning in fMRI-based human brain parcellation: a review

Li,

Chen,

Ling

et al. 2023

Meas. Sci. Technol.

View full text Add to dashboard Cite

Functional magnetic resonance imaging (fMRI)-based human brain parcellation reveals brain fundamental organizational principles noninvasively, providing prior guidance for functional analysis and physiological measurement of the brain. Recently, the profound success of deep learning in natural and medical images has attracted an increasing number of deep learning-based investigations for brain parcellation which have accomplished encouraging achievements. This review aims to provide researchers with a comprehensive overview of deep learning-based fMRI brain parcellation and promote the development of this emerging frontier. To begin, we present the history of brain parcellation, emphasizing the current advantages of deep learning in this field. Following a summary of the input modalities, traditional methods, and evaluation criteria for fMRI-based brain parcellation, we comprehensively review the current deep-learning approaches and categorize them based on network architectures. The datasets, models, performance, advantages and limitations are described in detail. Finally, we discuss the current challenges and future directions in methodology, neuroscience, and application, to inspire future investigation in deep learning-based brain parcellation.

show abstract

Deep multimodal fusion of image and non-image data in disease diagnosis and prognosis: a review

Cited by 49 publications

References 109 publications

An Ensemble Learning Approach for Multi-Modal Medical Image Fusion using Deep Convolutional Neural Networks

An Ensemble Learning Approach for Multi-Modal Medical Image Fusion using Deep Convolutional Neural Networks

Surviving ChatGPT in healthcare

Application of deep learning in fMRI-based human brain parcellation: a review

Contact Info

Product

Resources

About