Associating Textual Features with Visual Ones to Improve Affective Image Classification

2014 IEEE 16th International Workshop on Multimedia Signal Processing (MMSP)

Lau

Tjondronegoro

et al. 2014

The proliferation of news reports published in online websites and news information sharing among social media users necessitates effective techniques for analysing the image, text and video data related to news topics. This paper presents the first study to classify affective facial images on emerging news topics. The proposed system dynamically monitors and selects the current hot (of great interest) news topics with strong affective interestingness using textual keywords in news articles and social media discussions. Images from the selected hot topics are extracted and classified into three categorized emotions, positive, neutral and negative, based on facial expressions of subjects in the images. Performance evaluations on two facial image datasets collected from realworld resources demonstrate the applicability and effectiveness of the proposed system in affective classification of facial images in news reports. Facial expression shows high consistency with the affective textual content in news reports for positive emotion, while only low correlation has been observed for neutral and negative. The system can be directly used for applications, such as assisting editors in choosing photos with a proper affective semantic for a certain topic during news report preparation.

Section: Related Workmentioning

confidence: 99%

A pilot study on affective classification of facial images for emerging news topics

2014 IEEE 16th International Workshop on Multimedia Signal Processing (MMSP)

Lau

Tjondronegoro

et al. 2014

“…Generally, two types of methods are employed in existing studies on AVCA using affective dimensions: 1) constructing unsupervised models or knowledge-based rules to link the video content to affective dimensions by utilizing the knowledge learnt from associated fields such as psychology, art theory, and film [1], [2], [3], [4], and 2) employing supervised algorithms to learn the relationships between audiovisual data features and affective dimensions based on the training data [5], [6]. The first approach suffers from the drawback that the field knowledge is often subjective and may not fit for all types of real-world data.…”

Section: Introductionmentioning

confidence: 99%

Synchronous prediction of arousal and valence using LSTM network for affective video content analysis

2017 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD)

2017

The affect embedded in video data conveys high-level semantic information about the content and has direct impact on the understanding and perception of reviewers, as well as their emotional responses. Affective Video Content Analysis (AVCA) attempts to generate a direct mapping between video content and the corresponding affective states such as arousal and valence dimensions. Most existing studies establish the mapping for each dimension separately using knowledge-based rules or traditional classifiers such as Support Vector Machine (SVM). The inherent correlations between affective dimensions have largely been unexploited, which are anticipated to include important information for accurate prediction of affective dimensions. To address this issue, this paper presents an approach to predict arousal and valance dimensions synchronously using the Long Short Term Memory (LSTM) network. The approach extracts a set of low-level audio and visual features from video data and projects them synchronously into pairs of arousal and valence values using the LSTM network which automatically incorporates the correlations between arousal and valance dimensions. We evaluate the performance of the proposed approach on a dataset comprising video clips segmented from real-world resources such as film, drama, and news, and demonstrate its superior performance over the traditional SVM based method. The results provide one of the earliest preliminary evidence to the benefit of considering correlations between affective dimensions towards accurate AVCA.

“…While dramatic progress has been achieved in affective image classification [1,2,3], few attempts have been made to video emotion recognition. Compared to affective image analysis, video emotion analysis is a complex task.…”

Section: Introductionmentioning

confidence: 99%

Modeling temporal information using discrete fourier transform for recognizing emotions in user-generated videos

2016 IEEE International Conference on Image Processing (ICIP)

2016

With the widespread of user-generated Internet videos, emotion recognition in those videos attracts increasing research efforts. However, most existing works are based on framelevel visual features and/or audio features, which might fail to model the temporal information, e.g. characteristics accumulated along time. In order to capture video temporal information, in this paper, we propose to analyse features in frequency domain transformed by discrete Fourier transform (DFT features). Frame-level features are firstly extract by a pre-trained deep convolutional neural network (CNN). Then, time domain features are transferred and interpolated into DFT features. CNN and DFT features are further encoded and fused for emotion classification. By this way, static image features extracted from a pre-trained deep CNN and temporal information represented by DFT features are jointly considered for video emotion recognition. Experimental results demonstrate that combining DFT features can effectively capture temporal information and therefore improve emotion recognition performance. Our approach has achieved a stateof-the-art performance on the largest video emotion dataset (VideoEmotion-8 dataset), improving accuracy from 51.1% to 55.6%.