Affective Video Summarization and Story Board Generation Using Pupillary Dilation and Eye Gaze

Katti, Harish; Yadati, Karthik; Kankanhalli, Mohan; Chua, Tat-Seng

doi:10.1109/ism.2011.57

Cited by 33 publications

(32 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Shamir and colleagues [Shamir et al 2006] and Alves and colleagues [Alves et al 2008] provide algorithms to perform this selection based on text logs or scripts to summarize an interactive game play in comic book format. Katti and colleagues use gaze information, such as pupil dilation, to infer the portions of a video that are interesting to the viewers to create a storyboard or a comic book summary of a video [Katti et al 2011]. A recent work by Toyoura and colleagues also uses gaze information to detect scene changes and to insert word bubbles [Toyoura et al 2012b].…”

Section: Introductionmentioning

confidence: 99%

Inferring artistic intention in comic art through viewer gaze

Jain

Sheikh

Hodgins

2012

Proceedings of the ACM Symposium on Applied Perception

View full text Add to dashboard Cite

Comics are a compelling, though complex, visual storytelling medium. Researchers are interested in the process of comic art creation to be able to automatically tell new stories, and also, summarize videos and catalog large collections of photographs for example. A primary organizing principle used by artists to lay out the components of comic art (panels, word bubbles, objects inside each panel) is to lead the viewer's attention along a deliberate visual route that reveals the narrative. If artists are successful in leading viewer attention, then their intended visual route would be accessible through recorded viewer attention, i.e., eyetracking data. In this paper, we conduct an experiment to verify if artists are successful in their goal of leading viewer gaze. We eyetrack viewers on images taken from comic books, as well as photographs taken by experts, amateur photographers and a robot. Our data analyses show that there is increased consistency in viewer gaze for comic pictures versus photographs taken by a robot and by amateur photographers, thus confirming that comic artists do indeed direct the flow of viewer attention.

show abstract

Section: Introductionmentioning

confidence: 99%

Inferring artistic intention in comic art through viewer gaze

Jain

Sheikh

Hodgins

2012

Proceedings of the ACM Symposium on Applied Perception

View full text Add to dashboard Cite

show abstract

“…Relevant studies as reported in the literature- [14], [15] and [16] suggest that pupillary dilation gives reliable estimates of physiological arousal. According to [17], variations in the size of pupil also measures user's interest and engagement levels with the stimuli.…”

Section: Eye-tracking Experiment: Measuring Pupillary Dilation As mentioning

confidence: 99%

“…Similarly, a scene with an average score of below 3 on the two dimensions is labelled as a low arousal/valence scene. We compare the ground-truth arousal labels with the arousal labels generated using the pupillary dilation signal [17] and obtain an agreement level of 69%. We also compare the ground-truth arousal and valence labels with the labels generated using the content-based arousal and valence scores [7] and obtain an agreement level of 74% for arousal and 71% for valence.…”

Section: E Ground Truth Datamentioning

confidence: 99%

“…In the current eye-tracking experiment, we measure pupillary dilation as highlighted in Section IV-D. Since pupillary dilation measures user's interest and engagement levels [17], a new advertising strategy should induce similar or enhanced engagement levels, with the advertisements, when compared to the existing advertising strategies. In Fig.…”

Section: Eye-tracking Experimentsmentioning

confidence: 99%

See 1 more Smart Citation

CAVVA: Computational Affective Video-in-Video Advertising

Yadati

Katti

Kankanhalli³

2014

IEEE Trans. Multimedia

Self Cite

View full text Add to dashboard Cite

Abstract-Advertising is ubiquitous in the online community and more so in the ever-growing and popular online video delivery websites (e.g., YouTube). Video advertising is becoming increasingly popular on these websites. In addition to the existing pre-roll/post-roll advertising and contextual advertising, this paper proposes an in-stream video advertising strategy-Computational Affective Video-in-Video Advertising (CAVVA). Humans being emotional creatures are driven by emotions as well as rational thought. We believe that emotions play a major role in influencing the buying behavior of users and hence propose a video advertising strategy which takes into account the emotional impact of the videos as well as advertisements. Given a video and a set of advertisements, we identify candidate advertisement insertion points (step 1) and also identify the suitable advertisements (step 2) according to theories from marketing and consumer psychology. We formulate this two part problem as a single optimization function in a non-linear 0-1 integer programming framework and provide a genetic algorithm based solution. We evaluate CAVVA using a subjective user-study and eye-tracking experiment. Through these experiments, we demonstrate that CAVVA achieves a good balance between the following seemingly conflicting goals of (a) minimizing the user disturbance because of advertisement insertion while (b) enhancing the user engagement with the advertising content. We compare our method with existing advertising strategies and show that CAVVA can enhance the user's experience and also help increase the monetization potential of the advertising content.

show abstract

“…The process of key frame extraction is also known as "key-framing", "story-boarding" or "static video summarization". A video skim is a video of shorter length than the input stream, which is known as "dynamic video summarization" [1], [5], [6], [7], [8], [9].…”

Section: Introductionmentioning

confidence: 99%