Gaze-enabled egocentric video summarization via constrained submodular maximization

Xu, Jia; Mukherjee, Lopamudra; Li, Yin; Warner, Jamieson; Rehg, James M.; Singh, Vikas

doi:10.1109/cvpr.2015.7298836

Cited by 150 publications

(108 citation statements)

References 35 publications

Supporting

Mentioning

108

Contrasting

Order By: Relevance

“…Analysis of images captured by egocentric cameras can reveal a lot about the person recording such images, including, intentions, personality, interests, etc. Firstperson gaze prediction is useful in a wide range of applications in health care, education and entertainment, for tasks such as action and event recognition [35], recognition of handled objects [37], discovering important people [16], video re-editing [26], video summarization [45], engagement detection [42], and assistive vision systems [18].…”

Section: Introductionmentioning

confidence: 99%

Digging Deeper Into Egocentric Gaze Prediction

Tavakoli

Rahtu

Kannala

et al. 2019

2019 IEEE Winter Conference on Applications of Computer Vision (WACV)

View full text Add to dashboard Cite

This paper digs deeper into factors that influence egocentric gaze. Instead of training deep models for this purpose in a blind manner, we propose to inspect factors that contribute to gaze guidance during daily tasks. Bottom-up saliency and optical flow are assessed versus strong spatial prior baselines. Task-specific cues such as vanishing point, manipulation point, and hand regions are analyzed as representatives of top-down information. We also look into the contribution of these factors by investigating a simple recurrent neural model for ego-centric gaze prediction. First, deep features are extracted for all input video frames. Then, a gated recurrent unit is employed to integrate information over time and to predict the next fixation. We also propose an integrated model that combines the recurrent model with several top-down and bottom-up cues. Extensive experiments over multiple datasets reveal that (1) spatial biases are strong in egocentric videos, (2) bottom-up saliency models perform poorly in predicting gaze and underperform spatial biases, (3) deep features perform better compared to traditional features, (4) as opposed to hand regions, the manipulation point is a strong influential cue for gaze prediction, (5) combining the proposed recurrent model with bottom-up cues, vanishing points and, in particular, manipulation point results in the best gaze prediction accuracy over egocentric videos, (6) the knowledge transfer works best for cases where the tasks or sequences are similar, and (7) task and activity recognition can benefit from gaze prediction. Our findings suggest that (1) there should be more emphasis on hand-object interaction and (2) the egocentric vision community should consider larger datasets including diverse stimuli and more subjects.

show abstract

Section: Introductionmentioning

confidence: 99%

Digging Deeper Into Egocentric Gaze Prediction

Tavakoli

Rahtu

Kannala

et al. 2019

2019 IEEE Winter Conference on Applications of Computer Vision (WACV)

View full text Add to dashboard Cite

show abstract

“…In the supervised video summarization models, a key factor they are supposed to encompass is the diversity of the selected subset of video shots. This is often imposed by submodularity [10,16] and determinant [1,11,17]. When a video sequence is short, global diversity over the whole sequence seems like a natural choice [11,10].…”

Section: Introductionmentioning

confidence: 99%

How Local Is the Local Diversity? Reinforcing Sequential Determinantal Point Processes with Dynamic Ground Sets for Supervised Video Summarization

Wang

Yang

et al. 2018

Computer Vision – ECCV 2018

View full text Add to dashboard Cite

The large volume of video content and high viewing frequency demand automatic video summarization algorithms, of which a key property is the capability of modeling diversity. If videos are lengthy like hours-long egocentric videos, it is necessary to track the temporal structures of the videos and enforce local diversity. The local diversity refers to that the shots selected from a short time duration are diverse but visually similar shots are allowed to co-exist in the summary if they appear far apart in the video. In this paper, we propose a novel probabilistic model, built upon SeqDPP [1], to dynamically control the time span of a video segment upon which the local diversity is imposed. In particular, we enable SeqDPP to learn to automatically infer how local the local diversity is supposed to be from the input video. The resulting model is extremely involved to train by the hallmark maximum likelihood estimation (MLE), which further suffers from the exposure bias and non-differentiable evaluation metrics. To tackle these problems, we instead devise a reinforcement learning algorithm for training the proposed model. Extensive experiments verify the advantages of our model and the new learning algorithm over MLE-based methods.

show abstract

“…Such coherency can be performed by selecting sets of consecutive frames [5] or by adding temporal regularization [11]. The balance between interestingness and coherency, can be obtained using pre-segmentation methods [5], submodular optimization [6,19] or recurrent neural networks [22]. Here, we propose a model that jointly incorporates interestingness, coherency, and which is capable of adjusting the summaries based on a user-provided music-track.…”

Section: Related Workmentioning

confidence: 99%

Music-Guided Video Summarization using Quadratic Assignments

Mensink

Jongstra

Mettes

et al. 2017

Proceedings of the 2017 ACM on International Conference on Multimedia Retrieval

View full text Add to dashboard Cite

This paper aims to automatically generate a summary of an unedited video, guided by an externally provided music-track. The tempo, energy and beats in the music determine the choices and cuts in the video summarization. To solve this challenging task, we model video summarization as a quadratic assignment problem. We assign frames to the summary, using rewards based on frame interestingness, plot coherency, audio-visual match, and cut properties. Experimentally we validate our approach on the SumMe dataset. The results show that our music guided summaries are more appealing, and even outperform the current state-of-the-art summarization methods when evaluated on the F1 measure of precision and recall. CCS CONCEPTS INTROThe goal of this paper is to create high-quality video summarizations, guided by an externally provided music-track. Consider for example that after a day of skiing with your GoPro camera, you re ect your mood by selecting a music-track and the computer will automatically create a video summary of your skiing day tted on this speci c music-track. Clearly a summary with classical music should have di erent dynamics, plots, and cuts than a summary based on funk music, even when the summaries are created from Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for pro t or commercial advantage and that copies bear this notice and the full citation on the rst page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior speci c permission and/or a fee. Request permissions from permissions@acm.org. ICMR '17, Bucharest, Romania The key key factors used in our music-guided summarization model are illustrated in Fig. 1. We are inspired by a large body of research focused on video summarization, either using only the visual source video [5, 8, 11-13, 18, 22], or combining multiple video modalities [4,10,20]. In contrast to these works, we aim to create a video summary tted on a given music-track, which to the best of our knowledge has never been considered before.

show abstract

Gaze-enabled egocentric video summarization via constrained submodular maximization

Cited by 150 publications

References 35 publications

Digging Deeper Into Egocentric Gaze Prediction

Digging Deeper Into Egocentric Gaze Prediction

How Local Is the Local Diversity? Reinforcing Sequential Determinantal Point Processes with Dynamic Ground Sets for Supervised Video Summarization

Music-Guided Video Summarization using Quadratic Assignments

Contact Info

Product

Resources

About