Online Video Summarization: Predicting Future to Better Summarize Present

Lal, Shamit; Duggal, Shivam; Indu, S.

doi:10.1109/wacv.2019.00056

Cited by 26 publications

(10 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Aiming to better learn how to estimate the importance of video frames/fragments, some techniques pay attention to both the spatial and temporal structure of the video. [33] presents an Encoder-Decoder architecture with convolutional LSTMs that models the spatiotemporal relationship among parts of the video. [34] uses 3D-CNNs and convolutional LSTMs to model the spatiotemporal structure of the video and select the video key-frames, while [35] extracts spatial and temporal information by processing the raw frames and their optical flow maps with CNNs.…”

Section: A Supervised Video Summarizationmentioning

confidence: 99%

AC-SUM-GAN: Connecting Actor-Critic and Generative Adversarial Networks for Unsupervised Video Summarization

Apostolidis

Adamantidou

Metsai

et al. 2021

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

This paper presents a new method for unsupervised video summarization. The proposed architecture embeds an Actor-Critic model into a Generative Adversarial Network and formulates the selection of important video fragments (that will be used to form the summary) as a sequence generation task. The Actor and the Critic take part in a game that incrementally leads to the selection of the video key-fragments, and their choices at each step of the game result in a set of rewards from the Discriminator. The designed training workflow allows the Actor and Critic to discover a space of actions and automatically learn a policy for key-fragment selection. Moreover, the introduced criterion for choosing the best model after the training ends, enables the automatic selection of proper values for parameters of the training process that are not learned from the data (such as the regularization factor σ). Experimental evaluation on two benchmark datasets (SumMe and TVSum) demonstrates that the proposed AC-SUM-GAN model performs consistently well and gives SoA results in comparison to unsupervised methods, that are also competitive with respect to supervised methods.

show abstract

Section: A Supervised Video Summarizationmentioning

confidence: 99%

AC-SUM-GAN: Connecting Actor-Critic and Generative Adversarial Networks for Unsupervised Video Summarization

Apostolidis

Adamantidou

Metsai

et al. 2021

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

show abstract

“…Deep convolutional networks are typically used to extract features to describe individual video frames, and video shots can be represented using aggregations of frame-level representations such as the average feature vector [11] or representative vector [12]. Other methods use deep networks to directly predict the relevance of video frames [13] or deep recurrent networks to predict the relevance of entire video portions [11], [14], [15]. Some works create video skims by treating key-shot selection as object detection by using temporal region proposals to identify the potential locations of key-shots [10].…”

Section: A Video Summarizationmentioning

confidence: 99%

FCN-LectureNet: Extractive Summarization of Whiteboard and Chalkboard Lecture Videos

Davila

Setlur

et al. 2021

IEEE Access

View full text Add to dashboard Cite

Recording and sharing of educational or lecture videos has increased in recent years. Within these recordings, we find a large number of math-oriented lectures and tutorials which attract students of all levels. Many of the topics covered by these recordings are better explained using handwritten content on whiteboards or chalkboards. Hence, we find large numbers of lecture videos that feature the instructor writing on a surface. In this work, we propose a novel method for extraction and summarization of the handwritten content found in such videos. Our method is based on a fully convolutional network, FCN-LectureNet, which can extract the handwritten content from the video as binary images. These are further analyzed to identify the unique and stable units of content to produce a spatial-temporal index of handwritten content. A signal which approximates content deletion events is then built using information from the spatial-temporal index. The peaks of this signal are used to create temporal segments of the lecture based on the notion that sub-topics change when large portions of content are deleted. Finally, we use these segments to create an extractive summary of the handwritten content based on key-frames. This will facilitate content-based search and retrieval of these lecture videos. In this work, we also extend the AccessMath dataset to create a novel dataset for benchmarking of lecture video summarization called LectureMath. Our experiments on both datasets show that our novel method can outperform existing methods especially on the larger and more challenging dataset. Our code and data will be made publicly available.INDEX TERMS fully convolutional networks, handwritten text detection, image binarization, lecture videos, video summarization.

show abstract

“…Once again, the predicted importance scores are compared with the ground-truth data and the outcome of this comparison guides the training of the Summarizer. From this perspective, [37] presents an encoder-decoder architecture with convolu-tional LSTMs, that models the spatiotemporal relationship among parts of the video. In addition to the estimates about the frames' importance, the algorithm enhances the visual diversity of the summary via next frame prediction and shot detection mechanisms, based on the intuition that the first frames of a shot generally have high likelihood of being part of the summary.…”

Section: A Supervised Video Summarizationmentioning

confidence: 99%

“…Alternatively, they are extended by memory networks to increase the memorization capacity of the network and capture long-range temporal dependencies among parts of the video ( [35], [36]). Going one step further, a group of techniques try to learn importance by paying attention to both the spatial and temporal structure of the video, using convolutional LSTMs [37], [38], optical flow maps [39], combinations of CNNs and RNNs [40], or motion extraction mechanisms [42]. Following a different approach, a couple of supervised methods learn how to generate video summaries that are aligned with the human preferences the help of GANs ( [43], [44]).…”

Section: E General Remarks On Deep Learning Approachesmentioning

confidence: 99%

Video Summarization Using Deep Neural Networks: A Survey

Apostolidis¹,

Adamantidou²,

Metsai³

et al. 2021

Preprint

View full text Add to dashboard Cite

Video summarization technologies aim to create a concise and complete synopsis by selecting the most informative parts of the video content. Several approaches have been developed over the last couple of decades and the current state of the art is represented by methods that rely on modern deep neural network architectures. This work focuses on the recent advances in the area and provides a comprehensive survey of the existing deep-learning-based methods for generic video summarization. After presenting the motivation behind the development of technologies for video summarization, we formulate the video summarization task and discuss the main characteristics of a typical deep-learning-based analysis pipeline. Then, we suggest a taxonomy of the existing algorithms and provide a systematic review of the relevant literature that shows the evolution of the deep-learning-based video summarization technologies and leads to suggestions for future developments. We then report on protocols for the objective evaluation of video summarization algorithms and we compare the performance of several deep-learning-based approaches. Based on the outcomes of these comparisons, as well as some documented considerations about the suitability of evaluation protocols, we indicate potential future research directions.

show abstract

Online Video Summarization: Predicting Future to Better Summarize Present

Cited by 26 publications

References 11 publications

AC-SUM-GAN: Connecting Actor-Critic and Generative Adversarial Networks for Unsupervised Video Summarization

AC-SUM-GAN: Connecting Actor-Critic and Generative Adversarial Networks for Unsupervised Video Summarization

FCN-LectureNet: Extractive Summarization of Whiteboard and Chalkboard Lecture Videos

Video Summarization Using Deep Neural Networks: A Survey

Contact Info

Product

Resources

About