A Survey on Deep Learning for Multimodal Data Fusion

Gao, Jing; Li, Peng; Chen, Zhikui; Zhang, Jianing

doi:10.1162/neco_a_01273

Cited by 423 publications

(183 citation statements)

References 77 publications

Supporting

Mentioning

183

Contrasting

Order By: Relevance

“…For this purpose, the stateof-the-art deep learning models are powerful tools that have recently been used for combining and analyzing data from multiple sources together. Gao et al 2020 conducted a survey study on using deep learning technics for multimodal data fusion and how they can help find new interpretations of the data [28]. Thus, we predict that these deep learning models can be beneficial for multimodal data obtained from human studies as well.…”

Section: Discussionmentioning

confidence: 77%

Multimodal-Multisensory Experiments

Razavi¹,

Yamauchi²,

Janfaza³

et al. 2020

Preprint

View full text Add to dashboard Cite

The human mind is multimodal. Yet most behavioral studies rely on century-old measures of behavior—task accuracy and latency (response time). Multimodal and multisensory analysis of human behavior creates a better understanding of how the mind works. The problem is that designing and implementing these experiments is technically complex and costly. This paper introduces versatile and economical means of developing multimodal-multisensory human experiments. We provide an experimental design framework that automatically integrates and synchronizes measures including electroencephalogram (EEG), galvanic skin response (GSR), eye-tracking, virtual reality (VR), body movement, mouse/cursor motion and response time. Unlike proprietary systems (e.g., iMotions), our system is free and open-source; it integrates PsychoPy, Unity and Lab Streaming Layer (LSL). The system embeds LSL inside PsychoPy/Unity for the synchronization of multiple sensory signals—gaze motion, electroencephalogram (EEG), galvanic skin response (GSR), mouse/cursor movement, and body motion—with low-cost consumer-grade devices in a simple behavioral task designed by PsychoPy and a virtual reality environment designed by Unity. This tutorial shows a step-by-step process by which a complex multimodal-multisensory experiment can be designed and implemented in a few hours. When conducting the experiment, all of the data synchronization and recoding of the data to disk will be done automatically.

show abstract

Section: Discussionmentioning

confidence: 77%

Multimodal-Multisensory Experiments

Razavi¹,

Yamauchi²,

Janfaza³

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…To take full advantage of all available information in such studies, powerful multimodal deep learning methods are required. This has been well recognized in various other fields [258] , [259] , [303] but deserves more attention in bioimaging and calls for an integrative approach to bioimage analysis and bioinformatics.…”

Section: Discussionmentioning

confidence: 98%

A bird’s-eye view of deep learning in bioimage analysis

Meijering

2020

Computational and Structural Biotechnology Journal

115

View full text Add to dashboard Cite

Deep learning of artificial neural networks has become the de facto standard approach to solving data analysis problems in virtually all fields of science and engineering. Also in biology and medicine, deep learning technologies are fundamentally transforming how we acquire, process, analyze, and interpret data, with potentially far-reaching consequences for healthcare. In this mini-review, we take a bird’s-eye view at the past, present, and future developments of deep learning, starting from science at large, to biomedical imaging, and bioimage analysis in particular.

show abstract

“…The analysis of reviews [6,17,20,21,27,28] devoted to the problem of multimodal fusion allows us to highlight the main aspects that determine specific technological solutions for the fusion of images and texts in medicine:…”

Section: Background and Related Workmentioning

confidence: 99%

“…However, as noted in [47], this can be mainly due to resource reasons: the network for each unimodal stream can be designed and pre-trained independently for each modality. At the same time, late fusion can give rise to losing cross-modality information [27]. On the other hand, as the literature review reveals, the implementation of early and, even more so, multi-level fusion is a complex technological task and highly dependent on the subject area.…”

Section: Background and Related Workmentioning

confidence: 99%

Comparative assessment of text-image fusion models for medical diagnostics

Gusarova

Lobantsev

Vatian

et al. 2020

ICS

View full text Add to dashboard Cite

Introduction: Information overload and complexity are characteristic of decision-making in medicine. In these conditions information fusion techniques are effective. For the diagnosis and treatment of pneumonia using x-ray images and accompaniyng free-text radiologists reports, it is promising to use text-image fusion. Purpose: Development of a method for fusing text with an image in the treatment and diagnosis of pneumonia using neural networks. Methods: We used MIMIC-CXR dataset, the SEResNeXt101-32x4d for images feature extracting and the Bio-ClinicalBERT model followed by ContextLSTM layer for text featureextracting. We compared five architectures in the conducted experiment: image classifier, report classifier and three scenarios of the fusion, namely late fusion, middle fusion and early fusion. Results: We got an absolute excess of metrics (ROC AUC = 0.9933, PR AUC = 0.9907) when using an early fusion classifier (ROC AUC = 0.9921, PR AUC = 0.9889) even over the idealized case of text classifier (that is, without taking into account possible errors of the radiologist). The network training time ranged from 20 minutes for late fusion to 9 hours and 45 minutes for early fusion. Based on Class Activation Map technique we graphically showed that the image feature extractor in the fused classification scenario still learns discriminative regions for pneumonia classification problem. Discussion: Fusing text and images increases the likelihood of correct image classification compared to only image classification. The proposed combined image-report classifier trained with the early-fusion method gives better performance than individual classifiers in the pneumonia classification problem. However, it is worth considering that better results cost the training time and required computation resources. Report-based training is much faster in training and less demanding for computation capacity.

show abstract

A Survey on Deep Learning for Multimodal Data Fusion

Cited by 423 publications

References 77 publications

Multimodal-Multisensory Experiments

Multimodal-Multisensory Experiments

A bird’s-eye view of deep learning in bioimage analysis

Comparative assessment of text-image fusion models for medical diagnostics

Contact Info

Product

Resources

About