Saliency Tubes: Visual Explanations for Spatio-Temporal Convolutions

Stergiou, Alexandros; Kapidis, Georgios; Kalliatakis, Grigorios; Chrysoulas, Christos; Veltkamp, Remco C.; Poppe, Ronald

doi:10.1109/icip.2019.8803153

Cited by 39 publications

(23 citation statements)

References 28 publications

(36 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…At the same time egocentric videos usually offer a clear view of the camera Figure 1. Visualizing the class activation maps [45] for an instance of class 'open' from EPIC-Kitchens [7]. Left: Multi-Fiber Network (MFNet) [6] trained end-to-end for the single task of classifying short clips into actions.…”

Section: Introductionmentioning

confidence: 99%

Multitask Learning to Improve Egocentric Action Recognition

Kapidis

Poppe

Dam

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

Self Cite

View full text Add to dashboard Cite

In this work we employ multitask learning to capitalize on the structure that exists in related supervised tasks to train complex neural networks. It allows training a network for multiple objectives in parallel, in order to improve performance on at least one of them by capitalizing on a shared representation that is developed to accommodate more information than it otherwise would for a single task. We employ this idea to tackle action recognition in egocentric videos by introducing additional supervised tasks. We consider learning the verbs and nouns from which action labels consist of and predict coordinates that capture the hand locations and the gaze-based visual saliency for all the frames of the input video segments. This forces the network to explicitly focus on cues from secondary tasks that it might otherwise have missed resulting in improved inference. Our experiments on EPIC-Kitchens and EGTEA Gaze+ show consistent improvements when training with multiple tasks over the single-task baseline. Furthermore, in EGTEA Gaze+ we outperform the state-of-the-art in action recognition by 3.84%. Apart from actions, our method produces accurate hand and gaze estimations as side tasks, without requiring any additional input at test time other than the RGB video clips.

show abstract

Section: Introductionmentioning

confidence: 99%

Multitask Learning to Improve Egocentric Action Recognition

Kapidis

Poppe

Dam

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

Self Cite

View full text Add to dashboard Cite

show abstract

“…In this paper, in order to enable a hierarchical representation of kernel layer activations in 3D-CNNs, we propose Class Feature Pyramids for discovering kernel correspondence on the basis of how informative kernel activations are considered for classes. The visualization of kernel activations are based on the Saliency Tubes approach [43] which uses the spine interpolate of spatio-temporal kernel activations in order to create a representation conjoined with the used clip.…”

Section: Related Workmentioning

confidence: 99%

“…A number of previous works [7,43] have focused on creating representations of the regions in space and time that 3D-CNNs focus on when considering a particular class instance. These regions are class-specific and correspond to the activation maps produced in the last convolutional operation of the network.…”

Section: Discovery Of Prominent Spatio-temporal Feature Combinationsmentioning

confidence: 99%

Class Feature Pyramids for Video Explanation

Stergiou

Kapidis

Kalliatakis

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

Self Cite

View full text Add to dashboard Cite

Deep convolutional networks are widely used in video action recognition. 3D convolutions are one prominent approach to deal with the additional time dimension. While 3D convolutions typically lead to higher accuracies, the inner workings of the trained models are more difficult to interpret. We focus on creating human-understandable visual explanations that represent the hierarchical parts of spatiotemporal networks. We introduce Class Feature Pyramids, a method that traverses the entire network structure and incrementally discovers kernels at different network depths that are informative for a specific class. Our method does not depend on the network's architecture or the type of 3D convolutions, supporting grouped and depth-wise convolutions, convolutions in fibers, and convolutions in branches. We demonstrate the method on six state-of-the-art 3D convolution neural networks (CNNs) on three action recognition (Kinetics-400, UCF-101, and HMDB-51) and two egocentric action recognition datasets (EPIC-Kitchens and EGTEA Gaze+). 1

show abstract

“…To the best of our knowledge, this is the first method that enables such a direct feature correspondence. We extend the proposed regularisation method to enable visualisation with the inclusion of Saliency Tubes [241] at each block that it is applied to. Based on this, we create representations of features with the highest activations with respect to the selected class.…”

Section: Class Regularisation For Visual Explanationsmentioning

confidence: 99%

Efficient Modelling Across Time of Human Actions and Interactions

Stergiou¹

View full text Add to dashboard Cite

This thesis focuses on video understanding for human action and interaction recognition. We start by identifying the main challenges related to action recognition from videos and review how they have been addressed by current methods.Based on these challenges, and by focusing on the temporal aspect of actions, we argue that current fixed-sized spatio-temporal kernels in 3D convolutional neural networks (CNNs) can be improved to better deal with temporal variations in the input. Our contributions are based on the enlargement of the convolutional receptive fields through the introduction of spatio-temporal size-varying segments of videos, as well as the discovery of the local feature relevance over the entire video sequence. The resulting extracted features encapsulate information that includes the importance of local features across multiple temporal durations, as well as the entire video sequence.Subsequently, we study how we can better handle variations between classes of actions, by enhancing their feature differences over different layers of the architecture. The hierarchical extraction of features models variations of relatively similar classes the same as very dissimilar classes. Therefore, distinctions between similar classes are less likely to be modelled. The proposed approach regularises feature maps by amplifying features that correspond to the class of the video that is processed. We move away from class-agnostic networks and make early predictions based on feature amplification mechanism.The proposed approaches are evaluated on several benchmark action recognition datasets and show competitive results. In terms of performance, we compete with the state-of-the-art while being more efficient in terms of GFLOPs.Finally, we present a human-understandable approach aimed at providing visual explanations for features learned over spatio-temporal networks. We isolate spatio-temporal regions in 3D-CNNs that are informative for an action class. We extend this approach to allow for the traversal over the entire network architecture, incrementally discovering kernels at different complexities, and modelling layers related to a specific class.ix Chapter 1 IntroductionChapter 2, Related Work. We detail a synopsis of current progress made in action recognition. We distinguish between approaches that have been based on hand-crafted features and methods that used learned features through optimisation. We then present the main motivation for the works that have been included in this thesis. We discuss how our approaches differe from previous efforts and the challenges that our methods address.Chapter 3, Datasets for Video Understanding. We provide an overview of historic and current datasets used as action recognition benchmarks. We focus on datasets that exemplify milestones achieved in terms of data collection and increases in the data complexity. We then present the datasets that are used in this thesis.Chapter 4, Improving Action Recognition through Time-Consistent Features. We present a novel method to address var...

show abstract

Saliency Tubes: Visual Explanations for Spatio-Temporal Convolutions

Cited by 39 publications

References 28 publications

Multitask Learning to Improve Egocentric Action Recognition

Multitask Learning to Improve Egocentric Action Recognition

Class Feature Pyramids for Video Explanation

Efficient Modelling Across Time of Human Actions and Interactions

Contact Info

Product

Resources

About