A salient dictionary learning framework for activity video summarization via key-frame extraction

Mademlis, Ioannis; Tefas, Anastasios; Pitas, Ioannis

doi:10.1016/j.ins.2017.12.020

Cited by 49 publications

(18 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…L dict exploits h and a common matrix A. It is inspired by the dictionary-ofrepresentatives formulation of unsupervised video key-frame extraction [3]. In this generic framework, given original input video X ∈ R M ×T , the goal is to find an optimal summary matrix S ∈ R M ×C , C << T and a reconstruction coefficient matrix B ∈ R C×T , so that the columns of S constitute a subset of columns of X and the following objective is minimized:…”

Section: Proposed Dictionary Lossmentioning

confidence: 99%

“…Typical key-frame selection criteria were summary diversity and reconstructive ability, the latter being a way of formalizing representativeness as the degree to which the key-frames are jointly able to visually reconstruct all original video frames. Additionally, in various unsupervised approaches, key-frame difference from its temporal neighbours or similar saliency criteria were also employed [3,4,5,6]. In certain cases these algorithms processed raw video frames, but typically they were fed manually crafted image/video features [7,8].…”

Section: Introductionmentioning

confidence: 99%

“…The latter is obtained as the final hidden state of a newly introduced, parallel LSTM encoder that independently processes the full/complete video. Before computing this difference within the loss term, the summary representation is first projected into the space defined by a common set of basis vectors that are being simultaneously learnt from the entire training dataset, thus serving as a global visual dictionary [3]. Thus, each summary representation is exhorted towards being a set of linear reconstruction coefficients that are jointly able to reproduce the corresponding original sequence representation.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Adversarial Unsupervised Video Summarization Augmented With Dictionary Loss

Kaseris

Mademlis

Pitas

2021

2021 IEEE International Conference on Image Processing (ICIP)

Self Cite

View full text Add to dashboard Cite

Automated unsupervised video summarization by key-frame extraction consists in identifying representative video frames, best abridging a complete input sequence, and temporally ordering them to form a video summary, without relying on manually constructed ground-truth key-frame sets. Stateof-the-art unsupervised deep neural approaches consider the desired summary to be a subset of the original sequence, composed of video frames that are sufficient to visually reconstruct the entire input. They typically employ a pre-trained CNN for extracting a vector representation per RGB video frame and a baseline LSTM adversarial learning framework for identifying key-frames. In this paper, to better guide the network towards properly selecting video frames that can faithfully reconstruct the original video, we augment the baseline framework with an additional LSTM autoencoder, which learns in parallel a fixed-length representation of the entire original input sequence. This is exploited during training, where a novel loss term inspired by dictionary learning is added to the network optimization objectives, further biasing key-frame selection towards video frames which are collectively able to recreate the original video. Empirical evaluation on two common public relevant datasets indicates highly favourable results.

show abstract

Section: Proposed Dictionary Lossmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Adversarial Unsupervised Video Summarization Augmented With Dictionary Loss

Kaseris

Mademlis

Pitas

2021

2021 IEEE International Conference on Image Processing (ICIP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…Deep Learning Methods: To overcome the limitations of conventional methods, more recent works focus on designing deep learning models to tackle the problem of key frame detection. Several supervised and unsupervised models have been proposed for key frame detection in videos which significantly boost the performance of various downstream tasks [1,4,14,17,[49][50][51][52][53][54][55][56]. Yang et al [50] first introduced the bidirectional long short term memory (Bi-LSTM) for automatically extracting the highlights (key frames) from videos.…”

Section: Key Frame Detectionmentioning

confidence: 99%

Self-Supervised Learning to Detect Key Frames in Videos

Gilani

Feng

Zhang

et al. 2020

Sensors

View full text Add to dashboard Cite

Detecting key frames in videos is a common problem in many applications such as video classification, action recognition and video summarization. These tasks can be performed more efficiently using only a handful of key frames rather than the full video. Existing key frame detection approaches are mostly designed for supervised learning and require manual labelling of key frames in a large corpus of training data to train the models. Labelling requires human annotators from different backgrounds to annotate key frames in videos which is not only expensive and time consuming but also prone to subjective errors and inconsistencies between the labelers. To overcome these problems, we propose an automatic self-supervised method for detecting key frames in a video. Our method comprises a two-stream ConvNet and a novel automatic annotation architecture able to reliably annotate key frames in a video for self-supervised learning of the ConvNet. The proposed ConvNet learns deep appearance and motion features to detect frames that are unique. The trained network is then able to detect key frames in test videos. Extensive experiments on UCF101 human action and video summarization VSUMM datasets demonstrates the effectiveness of our proposed method.

show abstract

“…The visual summary of videos has also been widely studied in particular to provide a smart scroll bar when streaming videos. The two most common frameworks are key frame selection [6], [7], [8], [9] and key sub-shot selection [10], [11]. Both frameworks are classically unsupervised and aim at finding clusters that best describe all frames.…”

Section: Introductionmentioning

confidence: 99%

ActionSpotter: Deep Reinforcement Learning Framework for Temporal Action Spotting in Videos

Vaudaux-Ruth

Chan-Hon-Tong

Achard³

2021

2020 25th International Conference on Pattern Recognition (ICPR)

View full text Add to dashboard Cite

Action spotting has recently been proposed as an alternative to action detection and key frame extraction. However, the current state-of-the-art method of action spotting requires an expensive ground truth composed of the search sequences employed by human annotators spotting actions -a critical limitation.In this article, we propose to use a reinforcement learning algorithm to perform efficient action spotting using only the temporal segments from the action detection annotations, thus opening an interesting solution for video understanding.Experiments performed on THUMOS14 and ActivityNet datasets show that the proposed method, named ActionSpotter, leads to good results and outperforms state-of-the-art detection outputs redrawn for this application. In particular, the spotting mean Average Precision on THUMOS14 is significantly improved from 59.7% to 65.6% while skipping 23% of video.

show abstract

A salient dictionary learning framework for activity video summarization via key-frame extraction

Cited by 49 publications

References 33 publications

Adversarial Unsupervised Video Summarization Augmented With Dictionary Loss

Adversarial Unsupervised Video Summarization Augmented With Dictionary Loss

Self-Supervised Learning to Detect Key Frames in Videos

ActionSpotter: Deep Reinforcement Learning Framework for Temporal Action Spotting in Videos

Contact Info

Product

Resources

About