Advances in multimodal data fusion in neuroimaging: Overview, challenges, and novel orientation

Zhang, Yudong; Dong, Zhengchao; Wang, Shuihua; Yu, Xi; Yao, Xujing; Zhou, Qinghua; Hu, Hua; Li, Min; Jiménez-Mesa, Carmen; Ramı́rez, Javier; Martínez, Francisco J.; Górriz, Juan Manuel

doi:10.1016/j.inffus.2020.07.006

Cited by 332 publications

(128 citation statements)

References 426 publications

(471 reference statements)

Supporting

Mentioning

128

Contrasting

Order By: Relevance

“…The motivation of batch normalization (BAN) is to solve the “internal covariant shift (ICS)”, which means the effect of randomness of the distribution of inputs to internal CNN layers during training. The existence of ICS will worsen the CNN's performance [ 34 , 35 ].…”

Section: Methodsmentioning

confidence: 99%

Covid-19 classification by FGCNet with deep feature fusion from graph convolutional network and convolutional neural network

et al. 2021

Self Cite

View full text Add to dashboard Cite

Highlights We analysed over 320 COVID-19 images and 320 healthy control images. We proposed an improved CNN to extract individual image-level features. We proposed to use GCN to extract relation-aware representations. We proposed a DFF technology to combine features from GCN and CNN. The proposed FCGNet gives better performance than 15 state-of-the-art approaches.

show abstract

Section: Methodsmentioning

confidence: 99%

Covid-19 classification by FGCNet with deep feature fusion from graph convolutional network and convolutional neural network

et al. 2021

Self Cite

View full text Add to dashboard Cite

show abstract

“…The analysis of reviews [6,17,20,21,27,28] devoted to the problem of multimodal fusion allows us to highlight the main aspects that determine specific technological solutions for the fusion of images and texts in medicine:…”

Section: Background and Related Workmentioning

confidence: 99%

“…Here, such areas are actively developing as predicting the patient's health based on genomic, transcriptomic, and lifestyle information of one [17] as well as predicting the development of certain diseases [18,19]. For instance, multimodal fusion in neuroimaging is actively developing these days [20][21][22][23][24]. It combines data from multiple imaging modalities, like positron emission tomography, computed tomography, and magnetic resonance imaging, to overcome the limitations of individual УПРАВЛЕНИЕ В МЕДИЦИНЕ И БИОЛОГИИ modalities.…”

Section: Introductionmentioning

confidence: 99%

Comparative assessment of text-image fusion models for medical diagnostics

Gusarova

Lobantsev

Vatian

et al. 2020

ICS

View full text Add to dashboard Cite

Introduction: Information overload and complexity are characteristic of decision-making in medicine. In these conditions information fusion techniques are effective. For the diagnosis and treatment of pneumonia using x-ray images and accompaniyng free-text radiologists reports, it is promising to use text-image fusion. Purpose: Development of a method for fusing text with an image in the treatment and diagnosis of pneumonia using neural networks. Methods: We used MIMIC-CXR dataset, the SEResNeXt101-32x4d for images feature extracting and the Bio-ClinicalBERT model followed by ContextLSTM layer for text featureextracting. We compared five architectures in the conducted experiment: image classifier, report classifier and three scenarios of the fusion, namely late fusion, middle fusion and early fusion. Results: We got an absolute excess of metrics (ROC AUC = 0.9933, PR AUC = 0.9907) when using an early fusion classifier (ROC AUC = 0.9921, PR AUC = 0.9889) even over the idealized case of text classifier (that is, without taking into account possible errors of the radiologist). The network training time ranged from 20 minutes for late fusion to 9 hours and 45 minutes for early fusion. Based on Class Activation Map technique we graphically showed that the image feature extractor in the fused classification scenario still learns discriminative regions for pneumonia classification problem. Discussion: Fusing text and images increases the likelihood of correct image classification compared to only image classification. The proposed combined image-report classifier trained with the early-fusion method gives better performance than individual classifiers in the pneumonia classification problem. However, it is worth considering that better results cost the training time and required computation resources. Report-based training is much faster in training and less demanding for computation capacity.

show abstract

“…The fusion of deep learned features is a very effective way to enhance the accuracy of a health-related system [7] [8] . Typically, the fusion can be at three levels: feature level, classification level, and decision level.…”

Section: Introductionmentioning

confidence: 99%

COVID-19 and Non-COVID-19 Classification using Multi-layers Fusion From Lung Ultrasound Images

2021

View full text Add to dashboard Cite

COVID-19 or related viral pandemics should be detected and managed without hesitation, since the virus spreads very rapidly. Often with insufficient human and electronic resources, patients need to be checked from stable patients using vital signs, radiographic photographs, or ultrasound images. Vital signs do not often offer the right outcome, and radiographic photos have a variety of other problems. Lung ultrasound (LUS) images can provide good screening without a lot of complications. This paper suggests a model of a convolutionary neural network (CNN) that has fewer learning parameters but can achieve strong accuracy. The model has five main blocks or layers of convolution connectors. A multi-layer fusion functionality of each block is proposed to improve the efficiency of the COVID-19 screening method utilizing the proposed model. Experiments are conducted using freely accessible LUS photographs and video datasets. The proposed fusion method has 92.5% precision, 91.8% accuracy, and 93.2% retrieval using the data collection. These efficiency metric levels are considerably higher than those used in any of the state-of-the-art CNN versions.

show abstract

Advances in multimodal data fusion in neuroimaging: Overview, challenges, and novel orientation

Cited by 332 publications

References 426 publications

Covid-19 classification by FGCNet with deep feature fusion from graph convolutional network and convolutional neural network

Covid-19 classification by FGCNet with deep feature fusion from graph convolutional network and convolutional neural network

Comparative assessment of text-image fusion models for medical diagnostics

COVID-19 and Non-COVID-19 Classification using Multi-layers Fusion From Lung Ultrasound Images

Contact Info

Product

Resources

About