Multimodal medical image fusion towards future research: A review

Ullah Khan, Sajid; Ahmad Khan, Mir; Azhar, Muhammad; Khan, Faheem; Lee, Youngmoon; Javed, Muhammad

doi:10.1016/j.jksuci.2023.101733

Cited by 1 publication

(1 citation statement)

References 83 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The deep learning-based methods do not require manually designed algorithms and rules by using data-driven technology to automatically learn complicated feature representations [22]. Zhang et al proposed a generalized image fusion framework IFCNN based on convolutional neural networks (CNNs) [23].…”

Section: Multi-modal Medical Image Fusion Methodsmentioning

confidence: 99%

MDC-RHT: Multi-Modal Medical Image Fusion via Multi-Dimensional Dynamic Convolution and Residual Hybrid Transformer

Wang,

He,

Liu

et al. 2024

Sensors

View full text Add to dashboard Cite

The fusion of multi-modal medical images has great significance for comprehensive diagnosis and treatment. However, the large differences between the various modalities of medical images make multi-modal medical image fusion a great challenge. This paper proposes a novel multi-scale fusion network based on multi-dimensional dynamic convolution and residual hybrid transformer, which has better capability for feature extraction and context modeling and improves the fusion performance. Specifically, the proposed network exploits multi-dimensional dynamic convolution that introduces four attention mechanisms corresponding to four different dimensions of the convolutional kernel to extract more detailed information. Meanwhile, a residual hybrid transformer is designed, which activates more pixels to participate in the fusion process by channel attention, window attention, and overlapping cross attention, thereby strengthening the long-range dependence between different modes and enhancing the connection of global context information. A loss function, including perceptual loss and structural similarity loss, is designed, where the former enhances the visual reality and perceptual details of the fused image, and the latter enables the model to learn structural textures. The whole network adopts a multi-scale architecture and uses an unsupervised end-to-end method to realize multi-modal image fusion. Finally, our method is tested qualitatively and quantitatively on mainstream datasets. The fusion results indicate that our method achieves high scores in most quantitative indicators and satisfactory performance in visual qualitative analysis.

show abstract

Section: Multi-modal Medical Image Fusion Methodsmentioning

confidence: 99%