Robust Deep Multi-modal Learning Based on Gated Information Fusion Network

Kim, Jaekyum; Koh, Junho; Kim, Yecheol; Choi, Jaehyung; Hwang, Youngbae; Choi, Jun Won

doi:10.48550/arxiv.1807.06233

Cited by 3 publications

(3 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…for combining modalities [27,9,30], and recently it has been shown that variations of attention and gating mechanisms [16,2,32] can adapt to dynamic environments by weighting modalities differently for conditions that occur in the training data. While most works [21,31] have considered external environmental degradations such as rain, snow, glare, low-lighting, and seasonal appearance changes, more recent works [17] address robustness to different types of internal image degradations such as Gaussian noise.…”

Section: Related Work Fusion Architectures For Rgb-d Semantic Segment...mentioning

confidence: 99%

UNO: Uncertainty-aware Noisy-Or Multimodal Fusion for Unanticipated Input Degradation

Tian

Cheung

Glaser

et al. 2019

Preprint

View full text Add to dashboard Cite

The fusion of multiple sensor modalities, especially through deep learning architectures, has been an active area of study. However, an under-explored aspect of such work is whether the methods can be robust to degradations across their input modalities, especially when they must generalize to degradations not seen during training. In this work, we propose an uncertainty-aware fusion scheme to effectively fuse inputs that might suffer from a range of known and unknown degradations. Specifically, we analyze a number of uncertainty measures, each of which captures a different aspect of uncertainty, and we propose a novel way to fuse degraded inputs by scaling modalityspecific output softmax probabilities. We additionally propose a novel data-dependent spatial temperature scaling method to complement these existing uncertainty measures. Finally, we integrate the uncertainty-scaled output from each modality using a probabilistic noisy-or fusion method. In a photo-realistic simulation environment (AirSim), we show that our method achieves significantly better results on a semantic segmentation task, compared to state-of-art fusion architectures, on a range of degradations (e.g. fog, snow, frost, and various other types of noise), some of which are unknown during training. We specifically improve upon the state-of-art [30] by 28% in mean IoU on various degradations.

show abstract

Section: Related Work Fusion Architectures For Rgb-d Semantic Segment...mentioning

confidence: 99%

UNO: Uncertainty-aware Noisy-Or Multimodal Fusion for Unanticipated Input Degradation

Tian

Cheung

Glaser

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Part to be attention in each modal is acquired by a network itself. For example, in the case of images, the pixel of interest in the image is used [25].…”

Section: Multi-modal Learningmentioning

confidence: 99%

Deep Gated Multi-modal Learning: In-hand Object Pose Changes Estimation using Tactile and Image Data

Anzai¹,

Takahashi²

2019

Preprint

View full text Add to dashboard Cite

In robot manipulation tasks, especially in-hand manipulation, estimation of the position and orientation of an object is an essential skill to manipulate objects freely. However, since in-hand manipulation tends to cause occlusion by the hand itself, image information only is not sufficient. For the challenge, combining tactile sensors is one of the approaches. The advantage of using multiple sensors (modals) is that the other modals can compensate for occlusion, noise, and sensor malfunctions. Even though the decision making of each modal reliability corresponding to the situations is important, the manual design of the model is difficult to deal with various situations. Therefore, in this study, we propose deep gated multi-modal learning using end-to-end deep learning in which the network self-determines the reliability of each modal. As experiments, an RGB camera and a GelSight tactile sensor were attached to the gripper of the Sawyer robot, and the poses were estimated during grasping. A total of 15 objects were used in the experiments. In the proposed model, the reliability of the modal was determined according to the noise and failure of each modal, and it was confirmed that the pose was estimated even for unknown objects. 1 * The starred authors are contributed equally.1 T. Anzai is associated with The University of Tokyo. This work is an achievement during part-time job at Preferred Networks.

show abstract

“…2) These multiple intermediate features are merged into a joint representation via a fusion strategy. Such a fusion approach is referred to as intermidiate fusion [11] because the powerful intermediate features obtained by deep neural networks (DNNs) are merged to construct the joint representation. Deep multimodal learning has been shown to achieve remarkable performance for many machine learning tasks, such as deep cross-modal hashing [12] and deep semantic multimodal hashing [13].…”

Section: Introductionmentioning

confidence: 99%

Modal-aware Features for Multimodal Hashing

Zeng,

Lai,

Chu

et al. 2019

Preprint

View full text Add to dashboard Cite

Many retrieval applications can benefit from multiple modalities, e.g., text that contains images on Wikipedia, for which how to represent multimodal data is the critical component. Most deep multimodal learning methods typically involve two steps to construct the joint representations: 1) learning of multiple intermediate features, with each intermediate feature corresponding to a modality, using separate and independent deep models; 2) merging the intermediate features into a joint representation using a fusion strategy. However, in the first step, these intermediate features do not have previous knowledge of each other and cannot fully exploit the information contained in the other modalities. In this paper, we present a modal-aware operation as a generic building block to capture the non-linear dependences among the heterogeneous intermediate features that can learn the underlying correlation structures in other multimodal data as soon as possible. The modal-aware operation consists of a kernel network and an attention network. The kernel network is utilized to learn the non-linear relationships with other modalities. Then, to learn better representations for binary hash codes, we present an attention network that finds the informative regions of these modal-aware features that are favorable for retrieval. Experiments conducted on three public benchmark datasets demonstrate significant improvements in the performance of our method relative to state-of-the-art methods.

show abstract

Robust Deep Multi-modal Learning Based on Gated Information Fusion Network

Cited by 3 publications

References 26 publications

UNO: Uncertainty-aware Noisy-Or Multimodal Fusion for Unanticipated Input Degradation

UNO: Uncertainty-aware Noisy-Or Multimodal Fusion for Unanticipated Input Degradation

Deep Gated Multi-modal Learning: In-hand Object Pose Changes Estimation using Tactile and Image Data

Modal-aware Features for Multimodal Hashing

Contact Info

Product

Resources

About