Video Summarization with Long Short-Term Memory

Zhang, Ke; Chao, Weilun; Sha, Fei; Grauman, Kristen

doi:10.1007/978-3-319-46478-7_47

Cited by 554 publications

(724 citation statements)

References 38 publications

Supporting

Mentioning

717

Contrasting

Unclassified

Order By: Relevance

“…Our results are also better than the recent work of [22], where 38.6% is reported using slightly di erent evaluation settings and on par with their method that uses extensive additional labeled data to train interestingness and hyper-parameters (41.8%). Further, the relatively stable average Best F1 for the di erent music-tracks, indicate that the weighting of the components is not music-track speci c per se.…”

Section: Music-guided Summariescontrasting

confidence: 47%

“…Such coherency can be performed by selecting sets of consecutive frames [5] or by adding temporal regularization [11]. The balance between interestingness and coherency, can be obtained using pre-segmentation methods [5], submodular optimization [6,19] or recurrent neural networks [22]. Here, we propose a model that jointly incorporates interestingness, coherency, and which is capable of adjusting the summaries based on a user-provided music-track.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Music-Guided Video Summarization using Quadratic Assignments

Mensink

Jongstra

Mettes

et al. 2017

Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

This paper aims to automatically generate a summary of an unedited video, guided by an externally provided music-track. The tempo, energy and beats in the music determine the choices and cuts in the video summarization. To solve this challenging task, we model video summarization as a quadratic assignment problem. We assign frames to the summary, using rewards based on frame interestingness, plot coherency, audio-visual match, and cut properties. Experimentally we validate our approach on the SumMe dataset. The results show that our music guided summaries are more appealing, and even outperform the current state-of-the-art summarization methods when evaluated on the F1 measure of precision and recall. CCS CONCEPTS INTROThe goal of this paper is to create high-quality video summarizations, guided by an externally provided music-track. Consider for example that after a day of skiing with your GoPro camera, you re ect your mood by selecting a music-track and the computer will automatically create a video summary of your skiing day tted on this speci c music-track. Clearly a summary with classical music should have di erent dynamics, plots, and cuts than a summary based on funk music, even when the summaries are created from Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for pro t or commercial advantage and that copies bear this notice and the full citation on the rst page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior speci c permission and/or a fee. Request permissions from permissions@acm.org. ICMR '17, Bucharest, Romania The key key factors used in our music-guided summarization model are illustrated in Fig. 1. We are inspired by a large body of research focused on video summarization, either using only the visual source video [5, 8, 11-13, 18, 22], or combining multiple video modalities [4,10,20]. In contrast to these works, we aim to create a video summary tted on a given music-track, which to the best of our knowledge has never been considered before.

show abstract

Section: Music-guided Summariescontrasting

confidence: 47%

Section: Related Workmentioning

confidence: 99%

Music-Guided Video Summarization using Quadratic Assignments

Mensink

Jongstra

Mettes

et al. 2017

Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

show abstract

“…For the SumMe dataset, we compare the proposed method with seven baselines, VSUMM [10], SumTransfer [7], SUM-GAN [13], LSTM [15], MSDS-CC [27], LLR-SDS [28], and Online Motion AE [9] as shown in Table III, from which we can see that both VSUMM [10] and SumTransfer [7] cannot provide satisfactory performance, as employ the handcrafted features instead of more discriminative descriptors and thus cannot provide satisfactory performance. Also, we can find that our approach yields the best performance.…”

Section: Methodsmentioning

confidence: 99%

“…As a whole, eleven baselines are employed here for comparison, including: STIMO [26], VSUMM [10], SumTransfer [7], SUM-GAN [13], SeqDPP [6], LSTM [15], TVSum [8], Li et al [14], MSDS-CC [27], LLR-SDS [28], and Online Motion AE [9]. The results from all of the baselines [26,10,6,7,13,14,15,8,27,28,9] are obtained from those reported in their papers.…”

Section: Comparison With State-of-the-art Methodsmentioning

confidence: 99%

“…Li et al [14] proposed a model to summarize the videos based on a weighted combination of the importance, representativeness, diversity and smoothness of the storylinemeasures. Zhang et al [15] estimated the range of inter-dependencies among frames using long-short term memory (LSTM) to conduct supervised learning for video summarization. However, training recurrent neural networks such as LSTM and GAN is not an easy task and a large annotated dataset is in general required to attain satisfactory performance.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Video Summarization: How to Use Deep-Learned Features Without a Large-Scale Dataset

Purwanto

Chen

Fang

et al. 2018

EasyChair Preprints

View full text Add to dashboard Cite

Abstract-This paper proposes a framework incorporating deep-learned features with the conventional machine learning models within which the objective function is optimized by using quadratic programming or quasi-Newton methods instead of an end-to-end deep learning approach which uses variants of stochastic gradient descent algorithms. A temporal segmentation algorithm is first scrutinized by using a learning to rank scheme to detect the abrupt changes of frame appearances in a video sequence. Afterward, a peak-searching algorithm, statisticssensitive non-linear iterative peak-clipping (SNIP), is employed to acquire the local maxima of the filtered video sequence after rank pooling, where each of the local maxima corresponds to a key frame in the video. Simulations show that the new approach outperforms the main state-of-the-art works on four public video datasets.

show abstract

Materials Research at Shanghai Jiao Tong University

Chen

Feng

2015

Advanced Materials

View full text Add to dashboard Cite

Transformer architectures have exhibited remarkable performance in image super-resolution (SR). Since the quadratic computational complexity of the self-attention (SA) in Transformer, existing methods tend to adopt SA in a local region to reduce overheads. However, the local design restricts the global context exploitation, which is critical for accurate image reconstruction. In this work, we propose the Recursive Generalization Transformer (RGT) for image SR, which can capture global spatial information and is suitable for high-resolution images. Specifically, we propose the recursive-generalization self-attention (RG-SA). It recursively aggregates input features into representative feature maps, and then utilizes cross-attention to extract global information. Meanwhile, the channel dimensions of attention matrices (query, key, and value) are further scaled for a better trade-off between computational overheads and performance. Furthermore, we combine the RG-SA with local self-attention to enhance the exploitation of the global context, and propose the hybrid adaptive integration (HAI) for module integration. The HAI allows the direct and effective fusion between features at different levels (local or global). Extensive experiments demonstrate that our RGT outperforms recent state-of-the-art methods.

show abstract

Video Summarization with Long Short-Term Memory

Cited by 554 publications

References 38 publications

Music-Guided Video Summarization using Quadratic Assignments

Music-Guided Video Summarization using Quadratic Assignments

Video Summarization: How to Use Deep-Learned Features Without a Large-Scale Dataset

Materials Research at Shanghai Jiao Tong University

Contact Info

Product

Resources

About