ERP/MMR Algorithm for Classifying Topic‐Relevant and Topic‐Irrelevant Visual Shots of Documentary Videos

Abstract

We propose and evaluate a video summarization method based on a topic relevance model, a maximal marginal relevance (MMR), and discriminant analysis to generate a semantically meaningful video skim. The topic relevance model uses event‐related potential (ERP) components to describe the process of topic relevance judgment. More specifically, the topic relevance model indicates that N400 and P600, which have been successfully applied to the mismatch process of a stimulus and the discourse‐internal reorganization and integration process of a stimulus, respectively, are used for the topic mismatch process of a topic‐irrelevant video shot and the topic formation process of a topic‐relevant video shot. To evaluate our proposed ERP/MMR‐based method, we compared the video skims generated by the ERP/MMR‐based, ERP‐based, and shot boundary detection (SBD) methods with ground truth skims. The results showed that at a significance level of 0.05, the ROUGE‐1 scores of the ERP/MMR method are statistically higher than those of the SBD method, and the diversity scores of the ERP/MMR method are statistically higher than those of the ERP method. This study suggested that the proposed method may be applied to the construction of a video skim without operational intervention, such as the insertion of a black screen between video shots.

Revisiting Neurological Aspects of Relevance: An EEG Study

“…The positivity was significantly greater during the processing of 'rel' compared to the 'nr' content. Higher LPC amplitudes are associated with memory processing and decision-making, which might reflect effort invested in retaining relevant information during cumulative information exposure [22,51]. Furthermore, higher LPC amplitude deflection in response to task-related stimuli have previously been reported [5], which is consistent with our findings.…”

“…That 'nr' content is linked with less positivity in this time window bears similarity to the results of e.g. [22], who found that irrelevant content produced more negative N400 responses. 600 -750ms.…”

“…[20,38]) leading to the potential development of novel information search models [31] that can incorporate user's neurophysiological responses to the presented information. The most frequently used neuroimaging methods in the IR have been functional magnetic resonance (fMRI) [30-33, 36, 37], magnetoencephalography (MEG) [20] and electroencephalography (EEG) [2,17,22]. Within the relevance assessment context, the above-mentioned brain imaging techniques have been employed to investigate different aspects of neurological relevance assessment, such as the brain's functional connectivity [31], underlying cognitive processes and their timing [2].…”

“…Moshfeghi and colleagues [30] used fMRI to localise cortical activity differences during the processing of relevant vs non-relevant images that were related to visuospatial working memory [31,34]. Relevance assessment has also been studied within the context of the IR task for stimuli of different modalities, such as text [17], images [2] and videos [22]. Allegretti et al examined the processing of relevant vs non-relevant images, finding the most significant differences to occur between 500 -800ms, reaching the peak in the central scalp areas [2].…”

“…Allegretti et al examined the processing of relevant vs non-relevant images, finding the most significant differences to occur between 500 -800ms, reaching the peak in the central scalp areas [2]. Kim and Kim [22] explored the topical relevance of video skims and classified the data based on two specific ERP components (N400 and P600), which are indicators of relevant and non-relevant assessments. Furthermore, recent findings have shown that relevance assessment can be automatically predicted using EEG data while the user engages with the IR task [17].…”