EgoSampling: Wide View Hyperlapse From Egocentric Videos

Halperin, Tavi; Poleg, Yair; Arora, Chetan; Peleg, Bezalel

doi:10.1109/tcsvt.2017.2651051

Cited by 25 publications

(26 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…between sequence elements. This algorithm has been used extensively in the areas of speech processing [Hosom 2000;King et al 2012;Sakoe and Chiba 1978] and computer vision [Gong and Medioni 2011;Halperin et al 2018;Zhou et al 2008] for e.g. temporal segmentation and frame sampling, among many scientific disciplines.…”

Section: Related Workmentioning

confidence: 99%

Dynamic Temporal Alignment of Speech to Lips

Halperin

Ephrat

Peleg

2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

Fig. 1. Given a speech video and a segment of corresponding, but unaligned, audio, we align the audio to match the lip movements in the video.Many speech segments in movies are re-recorded in a studio during postproduction, to compensate for poor sound quality as recorded on location. Manual alignment of the newly-recorded speech with the original lip movements is a tedious task. We present an audio-to-video alignment method for automating speech to lips alignment, stretching and compressing the audio signal to match the lip movements. This alignment is based on deep audio-visual features, mapping the lips video and the speech signal to a shared representation. Using this shared representation we compute the lip-sync error between every short speech period and every video frame, followed by the determination of the optimal corresponding frame for each short sound period over the entire video clip. We demonstrate successful alignment both quantitatively, using a human perception-inspired metric, as well as qualitatively. The strongest advantage of our audio-to-video approach is in cases where the original voice in unclear, and where a constant shift of the sound can not give a perfect alignment. In these cases state-of-the-art methods will fail.

show abstract

Section: Related Workmentioning

confidence: 99%

Dynamic Temporal Alignment of Speech to Lips

Halperin

Ephrat

Peleg

2019

ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Self Cite

View full text Add to dashboard Cite

show abstract

“…• As expected, the coverage of any method goes down with the increase of hit number requirement. Nevertheless, for larger hit numbers (say, [10][11][12][13][14][15][16][17][18][19][20], our approach FFNet still outperforms all other methods. This shows its consistency in providing better coverage performance.…”

Section: Coverage Evaluationmentioning

confidence: 99%

“…However, these techniques require the processing of entire video and often take a long time to generate the subset. There are also video fast-forwarding techniques where the playback speed of the video is adjusted to meet the needs of users [3,10,13,31,32,34,38], but they often do not present an accurate representation and may still require processing of the entire video. Both types of approaches are not suitable for the resource-limited and time-critical systems we discussed above.…”

Section: Introductionmentioning

confidence: 99%

FFNet: Video Fast-Forwarding via Reinforcement Learning

Lan¹,

Panda

Zhu³

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

View full text Add to dashboard Cite

For many applications with limited computation, communication, storage and energy resources, there is an imperative need of computer vision methods that could select an informative subset of the input video for efficient processing at or near real time. In the literature, there are two relevant groups of approaches: generating a "trailer" for a video or fast-forwarding while watching/processing the video. The first group is supported by video summarization techniques, which require processing of the entire video to select an important subset for showing to users. In the second group, current fast-forwarding methods depend on either manual control or automatic adaptation of playback speed, which often do not present an accurate representation and may still require processing of every frame. In this paper, we introduce FastForwardNet (FFNet), a reinforcement learning agent that gets inspiration from video summarization and does fast-forwarding differently. It is an online framework that automatically fast-forwards a video and presents a representative subset of frames to users on the fly. It does not require processing the entire video, but just the portion that is selected by the fast-forward agent, which makes the process very computationally efficient. The online nature of our proposed method also enables the users to begin fast-forwarding at any point of the video. Experiments on two real-world datasets demonstrate that our method can provide better representation of the input video (about 6%-20% improvement on coverage of important frames) with much less processing requirement (more than 80% reduction in the number of frames processed).

show abstract

“…The nodes of this graph represent the frames of the original video and, the edges weights between pairs of frames are proportional to the cost of including the pair sequentially in the out-put video. An extension for creating a panoramic hyperlapse of a single or multiple input videos was proposed by Halperin et al [6]. They enlarge each of the input frames using neighboring frames from the videos to reduce the perception of instability.…”

Section: Related Workmentioning

confidence: 99%

A Weighted Sparse Sampling and Smoothing Frame Transition Approach for Semantic Fast-Forward First-Person Videos

Silva

Ramos

Ferreira

et al. 2018

2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition

View full text Add to dashboard Cite

Thanks to the advances in the technology of low-cost digital cameras and the popularity of the self-recording culture, the amount of visual data on the Internet is going to the opposite side of the available time and patience of the users. Thus, most of the uploaded videos are doomed to be forgotten and unwatched in a computer folder or website. In this work, we address the problem of creating smooth fast-forward videos without losing the relevant content. We present a new adaptive frame selection formulated as a weighted minimum reconstruction problem, which combined with a smoothing frame transition method accelerates first-person videos emphasizing the relevant segments and avoids visual discontinuities. The experiments show that our method is able to fast-forward videos to retain as much relevant information and smoothness as the state-ofthe-art techniques in less time. We also present a new 80hour multimodal (RGB-D, IMU, and GPS) dataset of firstperson videos with annotations for recorder profile, frame scene, activities, interaction, and attention 1 .

show abstract

EgoSampling: Wide View Hyperlapse From Egocentric Videos

Cited by 25 publications

References 33 publications

Dynamic Temporal Alignment of Speech to Lips

Dynamic Temporal Alignment of Speech to Lips

FFNet: Video Fast-Forwarding via Reinforcement Learning

A Weighted Sparse Sampling and Smoothing Frame Transition Approach for Semantic Fast-Forward First-Person Videos

Contact Info

Product

Resources

About