Learning with Privileged Information via Adversarial Discriminative Modality Distillation

Abstract

Heterogeneous data modalities can provide complementary cues for several tasks, usually leading to more robust algorithms and better performance. However, while training data can be accurately collected to include a variety of sensory modalities, it is often the case that not all of them are available in real life (testing) scenarios, where a model has to be deployed. This raises the challenge of how to extract information from multimodal data in the training stage, in a form that can be exploited at test time, considering limitations such as noisy or missing modalities. This paper presents a new approach in this direction for RGB-D vision tasks, developed within the adversarial learning and privileged information frameworks. We consider the practical case of learning representations from depth and RGB videos, while relying only on RGB data at test time. We propose a new approach to train a hallucination network that learns to distill depth information via adversarial learning, resulting in a clean approach without several losses to balance or hyperparameters. We report state-of-the-art results for object classification on the NYUD dataset, and video action recognition on the largest multimodal dataset available for this task, the NTU RGB+D, as well as on the Northwestern-UCLA.

Input Dropout for Spatially Aligned Modalities

“…In every case, Input Dropout performs better while being simpler than the other approaches. Note that we have not compared our approach to "modality distillation" [8] here since the method cannot be applied to this scenario. Indeed, it would involve training a network to dehaze a depth (or segmentation) image, which would require hallucinating scene contents.…”

“…We evaluate the use of Input Dropout for image classification using RGB+D training data. For this, we rely on the methodology proposed by Garcia et al [8], who use the crops of individual objects from the NYU V2 dataset [21] adapted by [13] for object classification using RGB+D. We used the same split as in [8]: 4,600 RGB-D images in total, where around 50% are used for training and the remainder for testing.…”

“…For this, we rely on the methodology proposed by Garcia et al [8], who use the crops of individual objects from the NYU V2 dataset [21] adapted by [13] for object classification using RGB+D. We used the same split as in [8]: 4,600 RGB-D images in total, where around 50% are used for training and the remainder for testing. Here, we rely on a ResNet-34 [12], initialized with pretrained weights on ImageNet [6].…”

“…Practical techniques have since been introduced, but those tend to be specifically targeted towards specific network architectures and applications. For example, "modality hallucination" [13] and variants [7,8] propose to train networks on different modalities independently, and shows that by changing the input modality of one of the networks while forcing the latent space to keep its former structure improves convergence. In [22], authors proposed to independently process multiple modalities in parallel branches within a network, and fuse the resulting feature maps using so-called "modality dropout" to make the network invariant to missing modalities.…”

“…In this paper, we propose a technique for exploiting additional modalities at training time, without having to rely on them at test time. In contrast to previous work which requires task-specific architectures [22] or multiple training passes [8], our approach is extremely simple (can be implemented in a few lines of code), is independent of the learning architecture used, and does not require any additional training pass. Assuming that modalities are spatially-aligned and share the same spatial resolution, we propose to randomly dropout [24] entire input modalities at training time.…”