Hahn-PCNN-CNN: an end-to-end multi-modal brain medical image fusion framework useful for clinical diagnosis

Abstract

Background In medical diagnosis of brain, the role of multi-modal medical image fusion is becoming more prominent. Among them, there is no lack of filtering layered fusion and newly emerging deep learning algorithms. The former has a fast fusion speed but the fusion image texture is blurred; the latter has a better fusion effect but requires higher machine computing capabilities. Therefore, how to find a balanced algorithm in terms of image quality, speed and computing power is still the focus of all scholars. Methods We built an end-to-end Hahn-PCNN-CNN. The network is composed of feature extraction module, feature fusion module and image reconstruction module. We selected 8000 multi-modal brain medical images downloaded from the Harvard Medical School website to train the feature extraction layer and image reconstruction layer to enhance the network’s ability to reconstruct brain medical images. In the feature fusion module, we use the moments of the feature map combined with the pulse-coupled neural network to reduce the information loss caused by convolution in the previous fusion module and save time. Results We choose eight sets of registered multi-modal brain medical images in four diease to verify our model. The anatomical structure images are from MRI and the functional metabolism images are SPECT and 18F-FDG. At the same time, we also selected eight representative fusion models as comparative experiments. In terms of objective quality evaluation, we select six evaluation metrics in five categories to evaluate our model. Conclusions The fusion image obtained by our model can retain the effective information in source images to the greatest extent. In terms of image fusion evaluation metrics, our model is superior to other comparison algorithms. In terms of time computational efficiency, our model also performs well. In terms of robustness, our model is very stable and can be generalized to multi-modal image fusion of other organs.

Effects of CT/MRI Image Fusion on Cerebrovascular Protection, Postoperative Complications, and Limb Functional Recovery in Patients with Anterior and Middle Skull Base Tumors: Based on a Retrospective Cohort Study

“…rough the dynamic guidance of intraoperative realtime fusion imaging technology, fusion imaging can locate the tumor more accurately and display the adjacent tumor in real time, which shortens the time for the operator to think about the location, scope, and adjacent structure of the tumor during the operation [34]. It reduces unnecessary brain tissue damage caused by exploration and search for tumors to speed up the postoperative recovery of the patients.…”

Multimodal Medical Image Fusion: The Perspective of Deep Learning

“…Wang et al [10] proposed a Gabor representation method that combines multiple CNNs and fuzzy neural networks to address the issue of inadequate representation of complex textures and edge information of lesions in fused multimodal medical images.Li et al [11] presented a multimodal medical image fusion approach that combines CNN and supervised learning. This method can handle various types of multimodal medical image fusion problems through batch processing, enhancing the overall efficiency and applicability of the technique.Guo et al [12] proposed an end-to-end CNN fusion framework based on a Pulse Coupled Neural Network (PCNN). In the feature fusion module, feature maps combined with a pulse coupled neural network were utilized to minimize information loss caused by convolution in the preceding fusion module, consequently improving computational efficiency.…”

WBC image classification and generative models based on convolutional neural network

“…Due to the vast success in a variety of applications, CNN has been adopted in several medical applications where imagery inputs are analyzed for diagnosis or classification. In the field of medical imaging, CNN has been successfully utilized for histological microscopic image [47], pediatric pneumonia [48], diabetic macular edema [48], ventricular arrhythmias [49], thyroid anomalies, mitotic nuclei estimation [50,51], neuroanatomy [52], and others [10][11][12][13][53][54][55][56][57][58][59]. Kermany et al [48] showed that CNN can detect diabetic macular edema and age-related macular degeneration with high accuracy and with a comparable performance of human experts.…”