Deep Robust Unsupervised Multi-Modal Network

Abstract

In real-world applications, data are often with multiple modalities, and many multi-modal learning approaches are proposed for integrating the information from different sources. Most of the previous multi-modal methods utilize the modal consistency to reduce the complexity of the learning problem, therefore the modal completeness needs to be guaranteed. However, due to the data collection failures, self-deficiencies, and other various reasons, multi-modal instances are often incomplete in real applications, and have the inconsistent anomalies even in the complete instances, which jointly result in the inconsistent problem. These degenerate the multi-modal feature learning performance, and will finally affect the generalization abilities in different tasks. In this paper, we propose a novel Deep Robust Unsupervised Multi-modal Network structure (DRUMN) for solving this real problem within a unified framework. The proposed DRUMN can utilize the extrinsic heterogeneous information from unlabeled data against the insufficiency caused by the incompleteness. On the other hand, the inconsistent anomaly issue is solved with an adaptive weighted estimation, rather than adjusting the complex thresholds. As DRUMN can extract the discriminative feature representations for each modality, experiments on real-world multimodal datasets successfully validate the effectiveness of our proposed method.

Complementary Fusion of Deep Network and Tree Model for ETA Prediction

“…(1) Dense features: Since ETA is a time prediction problem, the time-related features are particularly important, so we calculated the sum and maximum value of link-time and crosstime [9]. In addition, the state of road conditions is a key piece of information.…”

Noise-Aware Image Captioning with Progressively Exploring Mismatched Words

“…This involves comprehending the given image through computer vision techniques and generating corresponding descriptions using natural language processing. Initially, researchers explored the encoder-decoder architecture (Yang et al 2019a;Zhang et al 2019;Yang et al 2019c) with CNNs (Albawi, Mohammed, and Al-Zawi 2017) as image encoders and LSTM (Greff et al 2017) as text decoders (Vinyals et al 2015). To consider local and global features simultaneously, (Huang et al 2019) used image regions to decode image segmentations sequentially to words, adding attention mechanisms to focus on specific image regions during decoding.…”

A Region‐Based Analysis for the Feature Concatenation in Deep Forests

“…Later on, Feng and Zhou [24] show that random forests can do auto-encoder, implying that the informative rules of decision trees may accomplish representation learning. Deep forest is extended to numerous tasks and is successfully applied in metric learning [25] , multi-label learning [26] , semi-supervised learning [27] , financial fraud detection [28,29] , etc. Deep forests, on the other hand, require a significant amount of memory and time due to the storing of multi-layer forest modules to do layer-bylayer prediction on the test set.…”