Cross-modal Non-linear Guided Attention and Temporal Coherence in Multi-modal Deep Video Models

Abstract

Videos have data in multiple modalities, e.g., audio, video, text (captions). Understanding and modeling the interaction between different modalities is key for video analysis tasks like categorization, object detection, activity recognition, etc. However, data modalities are not always correlated-so, learning when modalities are correlated and using that to guide the influence of one modality on the other is crucial. Another salient feature of videos is the coherence between successive frames due to continuity of video and audio, a property that we refer to as temporal coherence. We show how using non-linear guided cross-modal signals and temporal coherence can improve the performance of multi-modal machine learning (ML) models for video analysis tasks like categorization. Our experiments on the large-scale YouTube-8M dataset show how our approach significantly outperforms state-of-the-art multi-modal ML models for video categorization. The model trained on the YouTube-8M dataset also showed good performance on an internal dataset of video segments from actual Samsung TV Plus channels without retraining or fine-tuning, showing the generalization capabilities of our model.

Disentangled Action Recognition with Knowledge Bases

“…Zero-shot learning with knowledge graphs: Zero-shot learning has been widely studied in computer vision (Akata et al, 2015;Lampert et al, 2013;Sahu et al, 2020;Xian et al, 2018). We will focus on related work relevant to our approach.…”

Enhancing Transformer for Video Understanding Using Gated Multi-Level Attention and Temporal Adversarial Training

“…We use YouTube-8M dataset for our experiments which consists of frame-wise video and audio features for approximately 5 million videos extracted using Inception v3 and VGGish respectively followed by PCA [1]. We use the hierarchical label space with 431 classes (see [17]). We use binary cross-entropy loss to train our models.…”

“…Existing works on video classification using deep learning can be broadly divided into four categories: (i) convolutional neural networks (CNNs) [21,12,4,20], (ii) recurrent neural networks (RNNs) [28,27,26], (iii) graph-based methods [13,2], and (iv) attention-based models [17,10,7].…”

“…Lastly, attention-based models use selfattention blocks at each layer to understand a frame's role with respect to other frames in the video. In this paper, we focus on attention approaches as they have low computation cost compared to 3-D CNNs and outperform RNN based approaches on video classification task [17].…”