Recent Advances and Trends in Multimodal Deep Learning: A Review

Abstract

Deep Learning has implemented a wide range of applications and has become increasingly popular in recent years. The goal of multimodal deep learning is to create models that can process and link information using various modalities. Despite the extensive development made for unimodal learning, it still cannot cover all the aspects of human learning. Multimodal learning helps to understand and analyze better when various senses are engaged in the processing of information. This paper focuses on multiple types of modalities, i.e., image, video, text, audio, body gestures, facial expressions, and physiological signals. Detailed analysis of past and current baseline approaches and an in-depth study of recent advancements in multimodal deep learning applications has been provided. A fine-grained taxonomy of various multimodal deep learning applications is proposed, elaborating on different applications in more depth. Architectures and datasets used in these applications are also discussed, along with their evaluation metrics. Last, main issues are highlighted separately for each domain along with their possible future research directions.CCS Concepts: • Computing methodologies → Machine learning; • Information systems → Multimedia and multimodal retrieval.

Getting the bugs out of AI: Advancing ecological research on arthropods through computer vision

“…Van Klink et al (Van Klink et al, 2022) recently highlighted how deep learning for ecology has been well represented in four distinct modalities: computer vision, acoustics, radar, and molecular methods. Recent successes in deep learning research have shown training models that utilize a combination of these representations can improve performances over a single modality, especially for fine-grained classification tasks (Morgado et al, 2021;Stahlschmidt et al, 2022;Summaira et al, 2021). We believe there are vast numbers of research directions to explore considering multimodal ecological data.…”

Artificial Intelligence-Assisted Renal Pathology: Advances and Prospects

“…To solve this problem, Vasiljević et al proposed an unsupervised learning algorithm to achieve “image translation” between different staining methods, which has proven useful in overcoming the barriers of inter-staining fusion [ 121 ]. Multi-modal learning, which refers to the combination of different sources of information [ 122 ], including pathological results, images, clinical history, and biochemical test results, has gained early success. An automated workflow was recently developed for integrating multi-modal data of mouse models of pancreatic cancer, which can transfer annotations between histology data and MSI data [ 123 ].…”

Getting the Bugs Out: Entomology Using Computer Vision

“…van Klink et al [17] recently highlighted how deep learning for ecology has been well represented in four distinct modalities: computer vision, acoustics, radar, and molecular methods. Recent successes in deep learning research have shown training models that utilize a combination of these representations can improve performances over a single modality, especially for fine-grained classification tasks [85,86,87]. We believe there are vast numbers of research directions to explore considering multimodal ecological data.…”