Divide-and-conquer based summarization framework for extracting affective video content

Abstract

Recent advances in multimedia technology have led to tremendous increases in the available volume of video data, thereby creating a major requirement for efficient systems to manage such huge data volumes. Video summarization is one of the key techniques for accessing and managing large video libraries. Video summarization can be used to extract the affective contents of a video sequence to generate a concise representation of its content. Human attention models are an efficient means of affective content extraction. Existing visual attention driven summarization frameworks have high computational cost and memory requirements, as well as a lack of efficiency in accurately perceiving human attention. To cope with these issues, we propose a divide-and-conquer based framework for an efficient summarization of big video data. We divide the original video data into shots, where an attention model is computed from each shot in parallel. Viewer attention is based on multiple sensory perceptions, i.e., aural and visual, as well as the viewer's neuronal signals. The aural attention model is based on the Teager energy, instant amplitude, and instant frequency, whereas the visual attention model employs multi-scale contrast and motion intensity. Moreover, the neuronal attention is computed using the beta-band frequencies of neuronal signals. Next, an aggregated attention curve is generated using an intra-and inter-modality fusion mechanism. Finally, the affective content in each video shot is extracted. The fusion of multimedia and neuronal signals provides a bridge that links the digital representation of multimedia with the viewer's perceptions. Our experimental results indicate that the proposed shot-detection based divide-and-conquer strategy mitigates the time and computational complexity. Moreover, the proposed attention model provides an accurate reflection of the user preferences and facilitates the extraction of highly affective and personalized summaries.

A combined multiple action recognition and summarization for surveillance video sequences

“…Each technique summarizes the video using a specific feature, such as trajectories, moving objects, abnormal detection, and many others. These categories of techniques can be classified into two general categories, scene-based (i.e., static [1-7, 9, 17, 21, 22], dynamic [8,12,13,16,[18][19][20] and content-based approaches, and the content-based approaches can be further decomposed into three types related to the content of the video including motion-based [10][11][12][13][14][15][16]20], action-based [21,22] and event-based [11,15,[17][18][19], as shown in Fig. 1.…”

“…Called HSUMM, the proposed approach adopts a hierarchical clustering method to generate a weight map from the frame similarity graph in which the clusters can easily be inferred. In the same context and to generate an efficient summarization, the authors in [8] proposed a divide-and-conquer-based framework. In this work, the original video data is divided into shots, where an attention model is computed from each shot in parallel.…”

“…From a video containing changing scenes, static-based methods consist of selecting keyframes and dynamic-based methods consist of selecting short video clips. Since the scenes can change, the cameras can move, the summarization, in this case, is carried out by determining the video sequences (shots) that represent the same scenes [8][9][10][11][12][13][14][15]. This allows the keyframe to be selected using extracted features and appropriate clustering methods.…”

Video summarization using event‐related potential responses to shot boundaries in real‐time video watching

“…The semantic gap refers to the gap between the actual semantic topic that a viewer infers from watching a video and the topic inferred from video summarization, which is constructed using the extraction algorithms. To bridge the semantic gap, visual‐attention‐model‐based summarization schemes have been proposed to extract frames as keyframes if they are visually important for humans, as determined using visual attention models (Ejaz, Mehmood, & Baik, ; Mehmood, Sajjad, Rho, & Baik, ). Mehmood et al () proposed a human‐attention model that combines both multimedia content and the viewer's neuronal responses for video summarization.…”

“…To bridge the semantic gap, visual‐attention‐model‐based summarization schemes have been proposed to extract frames as keyframes if they are visually important for humans, as determined using visual attention models (Ejaz, Mehmood, & Baik, ; Mehmood, Sajjad, Rho, & Baik, ). Mehmood et al () proposed a human‐attention model that combines both multimedia content and the viewer's neuronal responses for video summarization. In their model, neuronal attention is computed using beta‐band electroencephalography (EEG) frequencies of neuronal responses.…”