Query-adaptive Video Summarization via Quality-aware Relevance Estimation

Abstract

Although the problem of automatic video summarization has recently received a lot of a ention, the problem of creating a video summary that also highlights elements relevant to a search query has been less studied. We address this problem by posing queryrelevant summarization as a video frame subset selection problem, which lets us optimise for summaries which are simultaneously diverse, representative of the entire video, and relevant to a text query. We quantify relevance by measuring the distance between frames and queries in a common textual-visual semantic embedding space induced by a neural network. In addition, we extend the model to capture query-independent properties, such as frame quality. We compare our method against previous state of the art on textual-visual embeddings for thumbnail selection and show that our model outperforms them on relevance prediction. Furthermore, we introduce a new dataset, annotated with diversity and query-speci c relevance labels. On this dataset, we train and test our complete model for video summarization and show that it outperforms standard baselines such as Maximal Marginal Relevance.

Rethinking the Evaluation of Video Summaries

“…relationship between rankings). We take advantage of these tools in measuring the similarities between the implicit rankings provided by generated and human annotated frame level importance scores as in [20].…”

Personalized Video Summarization Based Exclusively on User Preferences

“…There exist supervised methods based on complex audiovisual features that can become personalized by training on annotations coming from a single user [18]. Other personalized methods use text queries [17]. They suffer, however, from the cold start problem, not being able to provide recommendations for users that are not in the training set.…”

Deep Reinforcement Learning for Query-Conditioned Video Summarization

“…The segment is thus regarded to be related to a query if it is either semantically similar to the query, or is visually similar to other segments which are related to the query. In [14], a quality-aware relevance estimation is proposed that relies on measuring the distance using the cosine similarity between embeddings of the frames and the text queries, where the textual representation is achieved by first using the word2vec model [33], and then an Long Short-Term Memory (LSTM) [34] to encode each into a single fixed-length embedding. Summaries are then obtained by selecting key frames using a linear combination of submodular objectives which jointly consider diversity, representativeness and quality of the visual features, as well as the similarity between the query and frames.…”

“…Different from traditional video summarization which only focuses on video content, query-conditioned video summarization is tasked to generate user-oriented summaries conditioned on a given query in the form of text [4,[12][13][14][15]. As shown in Figure 1, different user queries for the same video will have different summary results, so that the summary can be tailored to different user interests, leading to a better personalized summary.…”

“…It also enables more accurate evaluation by addressing the issue that generic video summarization is a highly subjective task where users may have different video-content preferences. Moreover, it enables users to more efficiently search video contents via text with multiple key words instead of only searching for video titles [14]. The illustration of the query-conditioned video summarization task.…”