Multi-task deep visual-semantic embedding for video thumbnail selection

Liu, Wu; Mei, Tao; Zhang, Yongdong; Che, Cherry; Luo, Jiebo

doi:10.1109/cvpr.2015.7298994

Cited by 220 publications

(130 citation statements)

References 19 publications

Supporting

Mentioning

130

Contrasting

Order By: Relevance

“…Of particular interest are applications of multimodal data using very deep networks. Recent research shows a deep semantic mapping between text and images [40] which motivates the use of knowledge extraction from, say, a text modality as relevant for the explanation of context in the image modality. Another direction for future work includes the parallel implementation of the proposed knowledge insertion and extraction algorithms, and its use in the iterative evaluation of very large networks, as part of a cycle of knowledge acquisition and revision.…”

Section: Discussionmentioning

confidence: 99%

Deep Logic Networks: Inserting and Extracting Knowledge From Deep Belief Networks

Tran

Garcez

2018

IEEE Trans. Neural Netw. Learning Syst.

102

View full text Add to dashboard Cite

This is the accepted version of the paper.This version of the publication may differ from the final published version. Abstract-Developments in deep learning have seen the use of layer-wise unsupervised learning combined with supervised learning for fine-tuning. With this layer-wise approach, a deep network can be seen as a more modular system which lends itself well to learning representations. In this paper we investigate whether such modularity can be useful to the insertion of background knowledge into deep networks, and whether it can improve learning performance when it is available, and to the extraction of knowledge from trained deep networks, and whether it can offer a better understanding of the representations learned by such networks. To this end we use a simple symbolic language -a set of logical rules which we call confidence rulesand show that it is suitable for the representation of quantitative reasoning in deep networks. We show by knowledge extraction that confidence rules can offer a low-cost representation for layerwise networks (or restricted Boltzmann machines). We also show that layer-wise extraction can produce an improvement in the accuracy of Deep Belief Networks. Furthermore, the proposed symbolic characterisation of deep networks provides a novel method for the insertion of prior knowledge and training of deep networks. With the use of this method, a deep neuralsymbolic system is proposed and evaluated, with experimental results indicating that modularity through the use of confidence rules and knowledge insertion can be beneficial to network performance. Permanent repository link

show abstract

Section: Discussionmentioning

confidence: 99%

Deep Logic Networks: Inserting and Extracting Knowledge From Deep Belief Networks

Tran

Garcez

2018

IEEE Trans. Neural Netw. Learning Syst.

102

View full text Add to dashboard Cite

show abstract

“…The matching relations between query words and frame contents are captured by a shallow dual cross-media relevance model [21] adapted from the image annotation problem. Liu et al [23] employed a deep visual-semantic embedding model (VSEM) to measure the relevance between the query and video frames by embedding them into a latent semantic space. Hence, key frames in the video are ranked by their distances to the given query in the learned latent space, and the top-ranked frames are selected as the final video thumbnail.…”

Section: Related Workmentioning

confidence: 99%

Sentence Specified Dynamic Video Thumbnail Generation

Yuan

Zhu

2019

Proceedings of the 27th ACM International Conference on Multimedia

View full text Add to dashboard Cite

With the tremendous growth of videos over the Internet, video thumbnails, providing video content previews, are becoming increasingly crucial to influencing users' online searching experiences. Conventional video thumbnails are generated once purely based on the visual characteristics of videos, and then displayed as requested. Hence, such video thumbnails, without considering the users' searching intentions, cannot provide a meaningful snapshot of the video contents that users concern. In this paper, we define a distinctively new task, namely sentence specified dynamic video thumbnail generation, where the generated thumbnails not only provide a concise preview of the original video contents but also dynamically relate to the users' searching intentions with semantic correspondences to the users' query sentences. To tackle such a challenging task, we propose a novel graph convolved video thumbnail pointer (GTP). Specifically, GTP leverages a sentence specified video graph convolutional network to model both the sentence-video semantic interaction and the internal video relationships incorporated with the sentence information, based on which a temporal conditioned pointer network is then introduced to sequentially generate the sentence specified video thumbnails. Moreover, we annotate a new dataset based on ActivityNet Captions for the proposed new task, which consists of 10,000+ videosentence pairs with each accompanied by an annotated sentence specified video thumbnail. We demonstrate that our proposed GTP outperforms several baseline methods on the created dataset, and thus believe that our initial results along with the release of the new dataset will inspire further research on sentence specified dynamic video thumbnail generation. Dataset and code are available at https://github.com/yytzsy/GTP. CCS CONCEPTS• Computing methodologies → Computer vision.

show abstract

“…In order to improve relevancy, Gao et al proposed to utilize semantic information to select semantically representative frames as video thumbnails [1]. Liu et al developed a multi-task deep visual-semantic embedding model to automatically select query-dependent thumbnails based on both semantic and visual features [2]. Both methods heavily depend on using semantic information to guarantee the representativeness of the selection.…”

Section: Automated Thumbnail Selectionmentioning

confidence: 99%

From Thumbnails to Summaries-A Single Deep Neural Network to Rule Them All

Swaminathan

2018

2018 IEEE International Conference on Multimedia and Expo (ICME)

View full text Add to dashboard Cite

Video summaries come in many forms, from traditional single-image thumbnails, animated thumbnails, storyboards, to trailer-like video summaries. Content creators use the summaries to display the most attractive portion of their videos; the users use them to quickly evaluate if a video is worth watching. All forms of summaries are essential to video viewers, content creators, and advertisers. Often video content management systems have to generate multiple versions of summaries that vary in duration and presentational forms. We present a framework ReconstSum that utilizes LSTM-based autoencoder architecture to extract and select a sparse subset of video frames or keyshots that optimally represent the input video in an unsupervised manner. The encoder selects a subset from the input video while the decoder seeks to reconstruct the video from the selection. The goal is to minimize the difference between the original input video and the reconstructed video. Our method is easily extendable to generate a variety of applications including static video thumbnails, animated thumbnails, storyboards and "trailer-like" highlights. We specifically study and evaluate two most popular use cases: thumbnail generation and storyboard generation. We demonstrate that our methods generate better results than the state-of-the-art techniques in both use cases.

show abstract

Multi-task deep visual-semantic embedding for video thumbnail selection

Cited by 220 publications

References 19 publications

Deep Logic Networks: Inserting and Extracting Knowledge From Deep Belief Networks

Deep Logic Networks: Inserting and Extracting Knowledge From Deep Belief Networks

Sentence Specified Dynamic Video Thumbnail Generation

From Thumbnails to Summaries-A Single Deep Neural Network to Rule Them All

Contact Info

Product

Resources

About