Multi-modal Representation Learning for Short Video Understanding and Recommendation

Abstract

Video advertisement content structuring aims to segment a given video advertisement and label each segment on various dimensions, such as presentation form, scene, and style. Different from real-life videos, video advertisements contain sufficient and useful multi-modal content like caption and speech, which provides crucial video semantics and would enhance the structuring process. In this paper, we propose a multi-modal encoder to learn multi-modal representation from video advertisements by interacting between video-audio and text. Based on multi-modal representation, we then apply Boundary-Matching Network to generate temporal proposals. To make the proposals more accurate, we refine generated proposals by scene-guided alignment and re-ranking. Finally, we incorporate proposal located embeddings into the introduced multimodal encoder to capture temporal relationships between local features of each proposal and global features of the whole video for classification. Experimental results show that our method achieves significantly improvement compared with several baselines and Rank 1 on the task of Multi-modal Ads Video Understanding in ACM Multimedia 2021 Grand Challenge. Ablation study further shows that leveraging multi-modal content like caption and speech in video advertisements significantly improve the performance.

Enriching Portuguese Word Embeddings with Visual Information

“…Also, when working with social networks, user data was often used as additional information when creating the embedding. User history was often used to cluster items that possess similar user bases [19,46].…”

Tamper-Proofing Video With Hierarchical Attention Autoencoder Hashing on Blockchain

“…The advent of deep learning delivered highly powerful hashing algorithms. Convolutional neural networks (CNN) were largely used to encode spatial content [13], [14], [15], [16]. Gou et al [13] used ResNet [17] to encode video and even audio features.…”

“…Convolutional neural networks (CNN) were largely used to encode spatial content [13], [14], [15], [16]. Gou et al [13] used ResNet [17] to encode video and even audio features. Xu et al [14] proposed to aggregate CNN frame features with VLAD encoding for event detection, sacrificing temporal coherence as the whole.…”

DeepFusion: Fusing User-Generated Content and Item Raw Content towards Personalized Product Recommendation

“…Multiple types of deep neural networks were utilized to analyze additional data sources including social networks, geospatial locations, and textual reviews. Guo et al [47] presented a multimodal representation learning method to predict user preferences based on multimodal content, including visual features, text features, audio features, and user interactive history in short video understanding and recommendation. Chen et al [48] proposed a novel neural architecture for fashion recommendation based on both image region-level features and user review information.…”