Multimodal Intelligence: Representation Learning, Information Fusion, and Applications

Abstract

Deep learning has revolutionized speech recognition, image recognition, and natural language processing since 2010, each involving a single modality in the input signal. However, many applications in artificial intelligence involve more than one modality. It is therefore of broad interest to study the more difficult and complex problem of modeling and learning across multiple modalities. In this paper, a technical review of the models and learning methods for multimodal intelligence is provided. The main focus is the combination of vision and natural language, which has become an important area in both computer vision and natural language processing research communities.This review provides a comprehensive analysis of recent work on multimodal deep learning from three new angles -learning multimodal representations, the fusion of multimodal signals at various levels, and multimodal applications. On multimodal representation learning, we review the key concept of embedding, which unifies the multimodal signals into the same vector space and thus enables cross-modality signal processing. We also review the properties of the many types of embedding constructed and learned for general downstream tasks. On multimodal fusion, this review focuses on special architectures for the integration of the representation of unimodal signals for a particular task. On applications, selected areas of a broad interest in current literature are covered, including caption generation, text-to-image generation, and visual question answering. We believe this review can facilitate future studies in the emerging field of multimodal intelligence for the community.

A Review on Explainability in Multimodal Deep Neural Nets

“…These diverse modalities differ in their scales, representation format, varied predictive power, weights, and contributions towards the final task [9]. Optimal data fusion schemes such as early [11], late [48], and hybrid fusion [49] schemes are developed to fuse the modalities at data, feature, decision, and intermediate mixed levels [50]. Deep neural nets [51],kernel-based methods [52], and graphical models [47,48] are employed for analysis and handling such data depending on the downstream task [46].…”

Multimodal Classification of Parkinson’s Disease in Home Environments with Resiliency to Missing Modalities

“…Multimodal Machine Learning —There is a long history of research in this area, exploring different directions [ 30 , 31 , 32 ]. Representation learning [ 33 , 34 , 35 ] is one of such directions in which effective and robust joint features are learned, typically from large-scale data sets, to be used in general downstream tasks, such as visual question answering or visual commonsense reasoning.…”

Unsupervised Feature Learning to Improve Transferability of Landslide Susceptibility Representations

“…Despite the growing amount of works on LSM, most of these methods have neglected the importance of utilizing the learned representation from well trained model, which causes them fail to transfer the learned knowledge to other datasets. Learning good representation from the input data is a core problem for unsupervised learning [12]. Although some recent studies [13] and [14] have shown good performance on extracting representation with impressive properties, there still exist some key problems needed to be fulfilled.…”