Visual-aural attention modeling for talk show video highlight detection

Zheng, Yijia; Zhu, Guangyu; Jiang, Shuqiang; Huang, Qingming; Gao, Wen

doi:10.1109/icassp.2008.4518084

Cited by 6 publications

(2 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Related efforts in recent years have assessed learners' emotions by using human physiological signals, such as heart rate variability (HRV) and EEG (Chen & Sun, 2012;Chen & Wang, 2011) and attention (Chen & Lin, 2014;Rebolledo-Mendez et al, 2009). Moreover, EEG signals have also been successfully applied in computer-based assessment (Wolpaw, McFarland, Neat & Forneris, 1991), brain-computer interface (Schalk, McFarland, Hinterberger, Birbaumer & Wolpaw, 2004;Wolpaw, Birbaumer, McFarland, Pfurtscheller & Vaughan, 2002), visual-aural attention modeling (Zheng et al, 2008), classification of human emotion (Murugappan, Nagarajan & Yaacob, 2010) and assessment of learning performance (Harmony et al, 2001). Of previous studies that developed AAS based on physiological signals, Hsu et al (2012) developed a reading concentration monitoring system to facilitate reading activity with e-books in order to allow instructors to more thoroughly understand students' reading concentration states.…”

Section: Literature Reviewmentioning

confidence: 99%

Assessing the attention levels of students by using a novel attention aware system based on brainwave signals

Chen¹,

Wang²,

Yu³

2015

Brit J Educational Tech

111

View full text Add to dashboard Cite

Rapid progress in information and communication technologies (ICTs) has fueled the popularity of e‐learning. However, an e‐learning environment is limited in that online instructors cannot monitor immediately whether students remain focus during online autonomous learning. Therefore, this study tries to develop a novel attention aware system (AAS) capable of recognizing students' attention levels accurately based on electroencephalography (EEG) signals, thus having high potential to be applied in providing timely alert for conveying low‐attention level feedback to online instructors in an e‐learning environment. To construct AAS, attention responses of students and their corresponding EEG signals are gathered based on a continuous performance test (CPT), ie, an attention assessment test. Next, the AAS is constructed by using training and testing data by the NeuroSky brainwave detector and the support vector machine (SVM), a well‐known machine learning model. Additionally, based on the discrete wavelet transform (DWT), the collected EEG signals are decomposed into five primary bands (ie, alpha, beta, gamma, theta, and delta). Each primary band contains five statistical parameters (including approximate entropy, total variation, energy, skewness, and standard deviation), thus generating 25 potential brainwave features associated with students' attention level for constructing the AAS. An attempt based on genetic algorithm (GA) is also made to enhance the prediction performance of the proposed AAS in terms of identifying students' attention levels. According to GA, the seven most influential features are selected from 25 considered features; parameters of the proposed AAS are also optimized. Analytical results indicate that the proposed AAS can accurately recognize individual student's attention state as either a high or low level, and the average accuracy rate reaches as high as 89.52%. Moreover, the proposed AAS is integrated with a video lecture tagging system to examine whether the proposed AAS can accurately detect students' low‐attention periods while learning about electrical safety in the workplace via a video lecture. Four experiments are designed to assess the prediction performance of the proposed AAS in terms of identifying the periods of video lecture with high‐ or low‐attention levels during learning processes. Analytical results indicate that the proposed AAS can accurately identify the low‐attention periods of video lecture generated by students when engaging in a learning activity with video lecture. Meanwhile, the proposed AAS can also accurately identify the low‐attention periods of video lecture generated by students to some degree even when students engage in a learning activity by a video lecture with random disturbances. Furthermore, strong negative correlations are found between the students' learning performance (ie, posttest score and progressive score) and the low‐attention periods of video lecture identified by the proposed AAS. Results of this study demonstrate that the proposed...

show abstract

Section: Literature Reviewmentioning

confidence: 99%

Assessing the attention levels of students by using a novel attention aware system based on brainwave signals

Chen¹,

Wang²,

Yu³

2015

Brit J Educational Tech

111

View full text Add to dashboard Cite

show abstract

“…Kayser proposed an audio attention computational model based on extracting contrast characters of intensity, duration and frequency from the Flourier spectrum in multi-scale to construct a bottom-up audio saliency map [5]. Zheng extracted audio features including short-term average energy, pitch, the average zero-crossing rate in his audio attention computational model to represent the strength of sounds, the sharpness of speeches and the degree of the urgency of audio [11]. Wang divided the audio signal into sub-frames to calculate the short-term energy, then, basing on the characteristics of auditory stream, energy spectrum of each channel was timedomain filtered in different scales through Gaussian filter groups and the auditory saliency map finally was achieved by linear combination of each frequency channel saliency [12].…”

Section: Related Workmentioning

confidence: 99%

An audio attention computational model based on information entropy of two channels and exponential moving average

Liu

Zhang

Hang

et al. 2019

Hum. Cent. Comput. Inf. Sci.

View full text Add to dashboard Cite

IntroductionAudio signal as the most important medium of human communication contains lots of information, so how to automatically detect some important area is the subject of attention by researchers from different fields. Different audio signal heard by human ear may cause our different level attention, and the processing of sounds heard by human ear is a series of complicating actions which are carried out to filter some information and the important content will be noticed by us in our daily environment. Both involuntary perception and voluntary perception are two modes of hearing the sound, the voluntary perception as a purposeful physiological behaviour based on prior knowledge and previous experience can help us clearly hear the sound in a very noisy environment, and the involuntary perception relies on the external environment and directly caused by a difference sound from the surrounding sound will give rise to pay more attention to the different sound in our environment. If we walk in a quiet park, a loud voice will immediately attract our attention. With no doubts, audio attention computational model used to AbstractThe main down-top attention model usually uses some characteristics of audio signal to extract the auditory saliency map at present. But existing audio attention computational model based image saliency mostly doesn't consider the continuity and attenuation mechanism of the human brain on paying attention to some occurred events in our real environment. To address these issues, we propose a model based on information entropy of two channels and exponential moving average (EMA) to simulate the processing of sound signals from the basilar membrane to the cochlear nucleus, and get the local information entropy of image and audio channel, finally we use the relevant EMA methods to calculate the auditory attention map. Some experimental results on artificial audio signal and real-world audio signal belonging to a public corpus show that the proposed model not only can detect the attention event, but also reflect the mechanism of human brain, and the experimental results of talk show in real environment are better. which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

show abstract

A Comprehensive Survey on Content Analysis and Its Challenges

Nayak

Dharmanna²

2020

Evolutionary Computing and Mobile Sustainable Networks

View full text Add to dashboard Cite

Visual-aural attention modeling for talk show video highlight detection

Cited by 6 publications

References 9 publications

Assessing the attention levels of students by using a novel attention aware system based on brainwave signals

Assessing the attention levels of students by using a novel attention aware system based on brainwave signals

An audio attention computational model based on information entropy of two channels and exponential moving average

A Comprehensive Survey on Content Analysis and Its Challenges

Contact Info

Product

Resources

About