Robust Deep Multi-modal Learning Based on Gated Information Fusion Network

Kim, Jaekyum; Koh, Junho; Kim, Yecheol; Choi, Jaehyung; Hwang, Youngbae; Choi, Jun Won

doi:10.1007/978-3-030-20870-7_6

Cited by 47 publications

(41 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…First, how to fuse the information. Sophisticated approaches use so-called gating networks which compute averaging weights in inference times, e.g., [ 24 , 25 ]. Another approach is to use recurrent networks to merge autoencoders [ 19 ].…”

Section: Related Workmentioning

confidence: 99%

Multi-Perspective Anomaly Detection

Jakob

Madan

Schmid-Schirling

et al. 2021

Sensors

View full text Add to dashboard Cite

Anomaly detection is a critical problem in the manufacturing industry. In many applications, images of objects to be analyzed are captured from multiple perspectives which can be exploited to improve the robustness of anomaly detection. In this work, we build upon the deep support vector data description algorithm and address multi-perspective anomaly detection using three different fusion techniques, i.e., early fusion, late fusion, and late fusion with multiple decoders. We employ different augmentation techniques with a denoising process to deal with scarce one-class data, which further improves the performance (ROC AUC =80%). Furthermore, we introduce the dices dataset, which consists of over 2000 grayscale images of falling dices from multiple perspectives, with 5% of the images containing rare anomalies (e.g., drill holes, sawing, or scratches). We evaluate our approach on the new dices dataset using images from two different perspectives and also benchmark on the standard MNIST dataset. Extensive experiments demonstrate that our proposed multi-perspective approach exceeds the state-of-the-art single-perspective anomaly detection on both the MNIST and dices datasets. To the best of our knowledge, this is the first work that focuses on addressing multi-perspective anomaly detection in images by jointly using different perspectives together with one single objective function for anomaly detection.

show abstract

Section: Related Workmentioning

confidence: 99%

Multi-Perspective Anomaly Detection

Jakob

Madan

Schmid-Schirling

et al. 2021

Sensors

View full text Add to dashboard Cite

show abstract

“…Fusion robustness is a rarely considered topic in the multi-modal fusion literature, however, it seems to be an important issue in real-world applications, especially in the robotics field. Only several papers [22,29,[31][32][33][34] took into account the non-nominal conditions of the multi-modal fusion and provided some analysis of fusion robustness to data degradation. Such degradation could occur due to sensor noise, its failure, or unexpected weather conditions.…”

Section: Fusion Robustnessmentioning

confidence: 99%

On Robustness of Multi-Modal Fusion—Robotics Perspective

2020

View full text Add to dashboard Cite

The efficient multi-modal fusion of data streams from different sensors is a crucial ability that a robotic perception system should exhibit to ensure robustness against disturbances. However, as the volume and dimensionality of sensory-feedback increase it might be difficult to manually design a multimodal-data fusion system that can handle heterogeneous data. Nowadays, multi-modal machine learning is an emerging field with research focused mainly on analyzing vision and audio information. Although, from the robotics perspective, haptic sensations experienced from interaction with an environment are essential to successfully execute useful tasks. In our work, we compared four learning-based multi-modal fusion methods on three publicly available datasets containing haptic signals, images, and robots’ poses. During tests, we considered three tasks involving such data, namely grasp outcome classification, texture recognition, and—most challenging—multi-label classification of haptic adjectives based on haptic and visual data. Conducted experiments were focused not only on the verification of the performance of each method but mainly on their robustness against data degradation. We focused on this aspect of multi-modal fusion, as it was rarely considered in the research papers, and such degradation of sensory feedback might occur during robot interaction with its environment. Additionally, we verified the usefulness of data augmentation to increase the robustness of the aforementioned data fusion methods.

show abstract

“…To implement an adaptively weighted fusion of multi-modal information in the intermediate feature levels so that the detector is robust to environmental changes, weighting mechanisms that determines the contribution of features from each modality are studied. Kim et al [47] employs the gated information fusion network to adjust the weights of the features from each modality so that degraded features are suppressed. 3D-CVF [5] develops a gated feature fusion network based on spatial-wise attention to generate the fused feature.…”

Section: Object Detection Based On Multi-modal Fusionmentioning

confidence: 99%

A Two-Phase Cross-Modality Fusion Network for Robust 3D Object Detection

Jiao

Yin

2020

Sensors

View full text Add to dashboard Cite

A two-phase cross-modality fusion detector is proposed in this study for robust and high-precision 3D object detection with RGB images and LiDAR point clouds. First, a two-stream fusion network is built into the framework of Faster RCNN to perform accurate and robust 2D detection. The visible stream takes the RGB images as inputs, while the intensity stream is fed with the intensity maps which are generated by projecting the reflection intensity of point clouds to the front view. A multi-layer feature-level fusion scheme is designed to merge multi-modal features across multiple layers in order to enhance the expressiveness and robustness of the produced features upon which region proposals are generated. Second, a decision-level fusion is implemented by projecting 2D proposals to the space of the point cloud to generate 3D frustums, on the basis of which the second-phase 3D detector is built to accomplish instance segmentation and 3D-box regression on the filtered point cloud. The results on the KITTI benchmark show that features extracted from RGB images and intensity maps complement each other, and our proposed detector achieves state-of-the-art performance on 3D object detection with a substantially lower running time as compared to available competitors.

show abstract

Robust Deep Multi-modal Learning Based on Gated Information Fusion Network

Cited by 47 publications

References 35 publications

Multi-Perspective Anomaly Detection

Multi-Perspective Anomaly Detection

On Robustness of Multi-Modal Fusion—Robotics Perspective

A Two-Phase Cross-Modality Fusion Network for Robust 3D Object Detection

Contact Info

Product

Resources

About