A Sparse Sampling-Based Framework for Semantic Fast-Forward of First-Person Videos

Abstract

Technological advances in sensors have paved the way for digital cameras to become increasingly ubiquitous, which, in turn, led to the popularity of the self-recording culture. As a result, the amount of visual data on the Internet is moving in the opposite direction of the available time and patience of the users. Thus, most of the uploaded videos are doomed to be forgotten and unwatched stashed away in some computer folder or website. In this paper, we address the problem of creating smooth fast-forward videos without losing the relevant content. We present a new adaptive frame selection formulated as a weighted minimum reconstruction problem. Using a smoothing frame transition and filling visual gaps between segments, our approach accelerates first-person videos emphasizing the relevant segments and avoids visual discontinuities. Experiments conducted on controlled videos and also on an unconstrained dataset of First-Person Videos (FPVs) show that, when creating fast-forward videos, our method is able to retain as much relevant information and smoothness as the state-of-the-art techniques, but in less processing time.

Musical Hyperlapse: A Multimodal Approach to Accelerate First-Person Videos

“…the Microsoft Hyperlapse (MSH) [Joshi et al, 2015], and the extended version of the Sparse Adaptive Sampling (SASv2) [Silva et al, 2021]. We used the desktop version of the MSH.…”

Text-Driven Video Acceleration: A Weakly-Supervised Reinforcement Learning Method

“…Silva et al [12] proposed modeling the frame sampling in semantic fast-forwarding as a Minimum Sparse Reconstruction problem. The authors also propose an extension [15] that aims to remove visual gaps that could break the continuity of the output video, and to smooth the speed-up transitions between video segments. A drawback of the aforementioned works consists of pre-processing steps like detecting objects and computing optical flow, which is timeconsuming and rely on the accuracy of third-party methods.…”

“…In our previous approach [14], we built an end-to-end trainable embedding space for text and image, which is further used by an RL agent to accelerate an input video. However, unlike most of the other fast-forward methods [5], [6], [8], [11], [12], [15], [40], [41], [42], the agent cannot optimize the output speed-up rate of the video, which is essential in several applications. In this work, we extend our previous approach [14] by introducing the Skip-Aware Fast-Forward Agent (SAFFA) and an Extended Visually-guided Document Attention Network (VDAN+).…”

“…We compared our method with the following fast-forwarding techniques: Sparse Adaptive Sampling (SAS) [12] and its extension SASv2 [15], which are state of the art in terms of semantics retained in the final video; the Bag-of-Topics (BoT) technique [13], which, like ours, also has texts and videos as input and; our previous approach [14] with the semantic encoder VDAN and the Semantic Fast-Forward Reinforcement Learning (SFF-RL) agent. We also report results for the Fast-Forward Network (FFNet), an RL agent proposed by Lan et al [10].…”

“…Over the past few years, algorithms on semantic fastforwarding have been emerging as effective approaches to tackle the task of retrieving meaningful segments without losing the temporal continuity [7], [8], [9], [10], [11], [12], [13], [14], [15]. In our previous work [14], we initially introduced an algorithm to create fast-forward videos guided by text documents.…”