Implicit video emotion tagging from audiences’ facial expression

Wang, Shangfei; Liu, Zhilei; Zhu, Yachen; He, Menghua; Chen, Xiaoping; Ji, Qiang

doi:10.1007/s11042-013-1830-0

Cited by 20 publications

(10 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Other methods include, a change in emotional intensity before and after watching videos [5], building user personality models [4], video affective content analysis including spontaneous response and emotional descriptors [3] etc. Furthermore, the effect of induced emotions in viewers plays a vital role in affective modeling and plenty of research on automatic machine generated predictions for ''intended elicited emotions in humans'' has although accomplished many milestones using audiovisual features [25], [26], implicit video emotion tagging using audiences' facial expression though Bayesian Network [65], affective image analysis based on perception subjectivity [67]- [70], scientific analysis of speech in phonetics such as in Praat [28], text-based extractions of tones and emotions such as IBM Watson's Tone Analyzer [27], but these techniques have certain drawbacks relative to approaches and techniques such as long-term context-aware information extraction, effect of continuous affective dynamics, reinforcing factors using implicit feedback mechanisms and real-time decision making using contextual information.…”

Section: Related Workmentioning

confidence: 99%

EmoWare: A Context-Aware Framework for Personalized Video Recommendation Using Affective Video Sequences

Tripathi¹,

Ashwin²,

Guddeti³

2019

IEEE Access

View full text Add to dashboard Cite

With the exponential growth in areas of machine intelligence, the world has witnessed promising solutions to the personalized content recommendation. The ability of interactive learning agents to make optimal decisions in dynamic environments has been proven and very well conceptualized by reinforcement learning (RL). The learning characteristics of deep-bidirectional recurrent neural networks (DBRNN) in both positive and negative time directions has shown exceptional performance as generative models to generate sequential data in supervised learning tasks. In this paper, we harness the potential of the said two techniques and propose EmoWare (emotion-aware), a personalized, emotionally intelligent video recommendation engine, employing a novel context-aware collaborative filtering approach, where the intensity of users' spontaneous non-verbal emotional response toward the recommended video is captured through interactions and facial expressions analysis for decision-making and video corpus evolution with real-time feedback streams. To account for users' multidimensional nature in the formulation of optimal policies, RL-scenarios are enrolled using on-policy (SARSA) and off-policy (Q-learning) temporal-difference learning techniques, which are used to train DBRNN to learn contextual patterns and to generate new video sequences for the recommendation. System evaluation for a month with real users shows that the EmoWare outperforms the state-of-the-art methods and models users' emotional preferences very well with stable convergence. INDEX TERMS Reinforcement learning, Q-learning, SARSA, deep bidirectoinal recurrent neural network, multi-armed bandit, video recommendation, affectiva, intensities of emotions, emotion-based information retrieval.

show abstract

Section: Related Workmentioning

confidence: 99%

EmoWare: A Context-Aware Framework for Personalized Video Recommendation Using Affective Video Sequences

Tripathi¹,

Ashwin²,

Guddeti³

2019

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Yoo and Cho [21] adopted an interactive genetic algorithm to realize individual scene video retrieval. Wang et al [90] employed a Bayesian network to capture the relationships between the video's common affective tag and the user's specific emotion label, and used the captured relationship to predict the video's personal tag. Soleymani et al [91] tackled the personal affective representation of movie scenes using the video's audiovisual features as well as physiological responses of participants to estimate the arousal and valence degree of scenes.…”

Section: Personalized Video Affective Content Analysismentioning

confidence: 99%

“…However, users' internal feelings and displayed facial behaviors are not always the same, evern though they are related [112], since expressions of emotion vary with person and context. Wang et al [90] employed a Bayesian network to capture the relationship between facial expressions and inner personalized emotions, and thus improved the performance of video emotion tagging.…”

Section: Implicit Video Affective Content Analysis Using Spontaneous mentioning

confidence: 99%

Video Affective Content Analysis: A Survey of State-of-the-Art Methods

Wang

2015

IEEE Trans. Affective Comput.

Self Cite

163

View full text Add to dashboard Cite

Video affective content analysis has been an active research area in recent decades, since emotion is an important component in the classification and retrieval of videos. Video affective content analysis can be divided into two approaches: direct and implicit. Direct approaches infer the affective content of videos directly from related audiovisual features. Implicit approaches, on the other hand, detect affective content from videos based on an automatic analysis of a user's spontaneous response while consuming the videos. This paper first proposes a general framework for video affective content analysis, which includes video content, emotional descriptors, and users' spontaneous nonverbal responses, as well as the relationships between the three. Then, we survey current research in both direct and implicit video affective content analysis, with a focus on direct video affective content analysis. Lastly, we identify several challenges in this field and put forward recommendations for future research.Index Terms-Video affective content analysis, emotion recognition, and content-based video retrieval.

show abstract

“…This problem refers to the association of tags to multimedia data based on the spontaneous reactions of users while watching the content. This reaction is measured automatically based on their facial expressions [145] or even based on their physiological reactions [65]. The wide availability of built-in sensors within devices capable of multimedia reproduction makes this an interesting and effortless way of tagging content.…”

Section: Assisting Behaviour Understanding Researchmentioning

confidence: 99%