Graph Distillation for Action Detection with Privileged Modalities

Luo, Zelun; Hsieh, Jun-Ting; Jiang, Lu; Niebles, Juan Carlos; Li, Feifei

doi:10.1007/978-3-030-01264-9_11

Cited by 103 publications

(75 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…al. [10] addressed a similar problem to ours, where the model is first trained on several modalities (RGB, depth, joints and infrared), but tested only in one. A graph-based distillation method able to distill information from all modalities at training time is proposed, while also passing through a validation phase on a subset of modalities.…”

Section: Related Workmentioning

confidence: 99%

Modality Distillation with Multiple Stream Networks for Action Recognition

Garcia

Morerio

Murino

2018

Lecture Notes in Computer Science

150

View full text Add to dashboard Cite

Diverse input data modalities can provide complementary cues for several tasks, usually leading to more robust algorithms and better performance. However, while a (training) dataset could be accurately designed to include a variety of sensory inputs, it is often the case that not all modalities could be available in real life (testing) scenarios, where a model has to be deployed. This raises the challenge of how to learn robust representations leveraging multimodal data in the training stage, while considering limitations at test time, such as noisy or missing modalities. This paper presents a new approach for multimodal video action recognition, developed within the unified frameworks of distillation and privileged information, named generalized distillation. Particularly, we consider the case of learning representations from depth and RGB videos, while relying on RGB data only at test time. We propose a new approach to train an hallucination network that learns to distill depth features through multiplicative connections of spatiotemporal representations, leveraging soft labels and hard labels, as well as distance between feature maps. We report state-of-the-art results on video action classification on the largest multimodal dataset available for this task, the NTU RGB+D. Code available at https://github.com/ncgarcia/ modality-distillation

show abstract

Section: Related Workmentioning

confidence: 99%

Modality Distillation with Multiple Stream Networks for Action Recognition

Garcia

Morerio

Murino

2018

Lecture Notes in Computer Science

150

View full text Add to dashboard Cite

show abstract

“…The performance values that refer to the Hoffman et al method [10] (row #20 of Table 4) are taken from the implementation and experiments in [11]. Row #21 refers to the method by Luo and colleagues [21], that uses 6 modalities at training time (RGB, depth, optical flow, and three different encoding methods for skeleton data), and RGB only at test time.…”

Section: Action Recognition Performance and Comparisonsmentioning

confidence: 99%

Learning with Privileged Information via Adversarial Discriminative Modality Distillation

Garcia

Morerio

Murino

2020

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Heterogeneous data modalities can provide complementary cues for several tasks, usually leading to more robust algorithms and better performance. However, while training data can be accurately collected to include a variety of sensory modalities, it is often the case that not all of them are available in real life (testing) scenarios, where a model has to be deployed. This raises the challenge of how to extract information from multimodal data in the training stage, in a form that can be exploited at test time, considering limitations such as noisy or missing modalities. This paper presents a new approach in this direction for RGB-D vision tasks, developed within the adversarial learning and privileged information frameworks. We consider the practical case of learning representations from depth and RGB videos, while relying only on RGB data at test time. We propose a new approach to train a hallucination network that learns to distill depth information via adversarial learning, resulting in a clean approach without several losses to balance or hyperparameters. We report state-of-the-art results for object classification on the NYUD dataset, and video action recognition on the largest multimodal dataset available for this task, the NTU RGB+D, as well as on the Northwestern-UCLA.

show abstract

“…At testing, it used the softmax to select the final prediction between the predictions from the hallucination representation and the predictions from RGB representation. Luo et al [15] recently proposed graph distillation for action detection with privileged modalities (RGB, depth, skeleton, and flow), where a novel graph distillation layer was used to dynamically learn to distill knowledge from the most effective modality, depending on the type of action. In our case, we use paired depth and RGB images during training.…”

Section: Related Workmentioning

confidence: 99%

3D Hand Pose Estimation from RGB Using Privileged Learning with Depth Data

Yuan

Stenger

Kim

2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

This paper proposes a method for hand pose estimation from RGB images that uses both external large-scale depth image datasets and paired depth and RGB images as privileged information at training time. We show that providing depth information during training significantly improves performance of pose estimation from RGB images during testing. We explore different ways of using this privileged information: (1) using depth data to initially train a depth-based network, (2) using the features from the depthbased network of the paired depth images to constrain midlevel RGB network weights, and (3) using the foreground mask, obtained from the depth data, to suppress the responses from the background area. By using paired RGB and depth images, we are able to supervise the RGB-based network to learn middle layer features that mimic that of the corresponding depth-based network, which is trained on large-scale, accurately annotated depth data. During testing, when only an RGB image is available, our method produces accurate 3D hand pose predictions. Our method is also tested on 2D hand pose estimation. Experiments on three public datasets show that the method outperforms the state-of-the-art methods for hand pose estimation using RGB image input.

show abstract

Graph Distillation for Action Detection with Privileged Modalities

Cited by 103 publications

References 45 publications

Modality Distillation with Multiple Stream Networks for Action Recognition

Modality Distillation with Multiple Stream Networks for Action Recognition

Learning with Privileged Information via Adversarial Discriminative Modality Distillation

3D Hand Pose Estimation from RGB Using Privileged Learning with Depth Data

Contact Info

Product

Resources

About