Representation of facial expression categories in continuous arousal–valence space: Feature and correlation

Zhang, Ligang; Tjondronegoro, Dian; Chandran, Vinod

doi:10.1016/j.imavis.2014.09.005

Cited by 20 publications

(10 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These approaches primarily generate the mapping between audio-visual features and each affective dimension separately, and thus the intrinsic correlations between affective dimensions are largely ignored. As previous studies [7] and our previous work [14] showed that there often exist strong correlations between discrete affective categories and affective dimensions, as well as between affective dimensions, it is desirable to exploit the usefulness of the correlations between affective dimensions in improving the prediction accuracy of AVCA. The incorporation of such correlations is also anticipated to be an effective way to reduce the affective gap [2], which is caused by the lack of coincidence between the measurable features and the expected affective states of the viewer.…”

Section: B Avca Approaches Using Affective Dimensionsmentioning

confidence: 96%

Synchronous prediction of arousal and valence using LSTM network for affective video content analysis

Zhang

2017

2017 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD)

Self Cite

View full text Add to dashboard Cite

The affect embedded in video data conveys high-level semantic information about the content and has direct impact on the understanding and perception of reviewers, as well as their emotional responses. Affective Video Content Analysis (AVCA) attempts to generate a direct mapping between video content and the corresponding affective states such as arousal and valence dimensions. Most existing studies establish the mapping for each dimension separately using knowledge-based rules or traditional classifiers such as Support Vector Machine (SVM). The inherent correlations between affective dimensions have largely been unexploited, which are anticipated to include important information for accurate prediction of affective dimensions. To address this issue, this paper presents an approach to predict arousal and valance dimensions synchronously using the Long Short Term Memory (LSTM) network. The approach extracts a set of low-level audio and visual features from video data and projects them synchronously into pairs of arousal and valence values using the LSTM network which automatically incorporates the correlations between arousal and valance dimensions. We evaluate the performance of the proposed approach on a dataset comprising video clips segmented from real-world resources such as film, drama, and news, and demonstrate its superior performance over the traditional SVM based method. The results provide one of the earliest preliminary evidence to the benefit of considering correlations between affective dimensions towards accurate AVCA.

show abstract

Section: B Avca Approaches Using Affective Dimensionsmentioning

confidence: 96%

Synchronous prediction of arousal and valence using LSTM network for affective video content analysis

Zhang

2017

2017 13th International Conference on Natural Computation, Fuzzy Systems and Knowledge Discovery (ICNC-FSKD)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The speech emotion recognition study [39] illustrates six categories of human vocal emotions. Among various facial emotion studies, [5,68] represent the facial expressions with seven categories in two-dimensional valencearousal space. A three-dimensional model for eight basic facial emotions and monoamine neurotransmitters is represented in [33] using the corners of a cube.…”

Section: Music Video Emotion Datasetmentioning

confidence: 99%

Deep learning-based late fusion of multimodal information for emotion classification of music video

Pandeya

Lee

2020

Multimed Tools Appl

127

View full text Add to dashboard Cite

Affective computing is an emerging area of research that aims to enable intelligent systems to recognize, feel, infer and interpret human emotions. The widely spread online and off-line music videos are one of the rich sources of human emotion analysis because it integrates the composer’s internal feeling through song lyrics, musical instruments performance and visual expression. In general, the metadata which music video customers to choose a product includes high-level semantics like emotion so that automatic emotion analysis might be necessary. In this research area, however, the lack of a labeled dataset is a major problem. Therefore, we first construct a balanced music video emotion dataset including diversity of territory, language, culture and musical instruments. We test this dataset over four unimodal and four multimodal convolutional neural networks (CNN) of music and video. First, we separately fine-tuned each pre-trained unimodal CNN and test the performance on unseen data. In addition, we train a 1-dimensional CNN-based music emotion classifier with raw waveform input. The comparative analysis of each unimodal classifier over various optimizers is made to find the best model that can be integrate into a multimodal structure. The best unimodal modality is integrated with corresponding music and video network features for multimodal classifier. The multimodal structure integrates whole music video features and makes final classification with the SoftMax classifier by a late feature fusion strategy. All possible multimodal structures are also combined into one predictive model to get the overall prediction. All the proposed multimodal structure uses cross-validation to overcome the data scarcity problem (overfitting) at the decision level. The evaluation results using various metrics show a boost in the performance of the multimodal architectures compared to each unimodal emotion classifier. The predictive model by integration of all multimodal structure achieves 88.56% in accuracy, 0.88 in f1-score, and 0.987 in area under the curve (AUC) score. The result suggests human high-level emotions are automatically well classified in the proposed CNN-based multimodal networks, even though a small amount of labeled data samples is available for training.

show abstract

“…Dimensional spaces have the advantage of representing a wide range of emotions, and can provide unique insights into the relationship between emotions and emotional intensity. It should be noted that most existing approaches to emotion recognition in dimensional spaces quantize the dimensions into a number of intervals, such as the four quadrants [23], or negative and positive emotions [24], and only few studies [25] have investigated continuous dimensions. This paper adopts three emotion categories of positive, neutral and negative for the classification experiments, catering for the aim for practical applications where facial expressions are more complicated than pre-defined emotions.…”

Section: Facial Expression Representationmentioning

confidence: 99%

“…Thus, the size of 8×10 is used for LBP feature extraction in the proposed system. [34,35,39,45,48,41,40,46,49,26] and [34,40,48,39,41,45,49,25,35,46], respectively. It is interesting to observe that the two databases share 9 out of the top 10 points (indices: [34,35,39,45,48,41,40,46,49]).…”

Section: Comparisons Of Key Parametersmentioning

confidence: 99%

Towards robust automatic affective classification of images using facial expressions for practical applications

Zhang

Tjondronegoro

Chandran

et al. 2015

Multimed Tools Appl

Self Cite

View full text Add to dashboard Cite

Affect is an important feature of multimedia content and conveys valuable information for multimedia indexing and retrieval. Most existing studies for affective content analysis are limited to low-level features or mid-level representations, and are generally criticized for their incapacity to address the gap between low-level features and high-level human affective perception. The facial expressions of subjects in images carry important semantic information that can substantially influence human affective perception, but have been seldom investigated for affective classification of facial images towards practical applications. This paper presents an automatic image emotion detector (IED) for affective classification of practical (or non-laboratory) data using facial expressions, where a lot of "real-world" challenges are present, including pose, illumination, and size variations. The proposed method is novel, with its framework designed specifically to overcome these challenges using multi-view versions of face and point detectors, and a combination of pointbased texture and geometry. Performance comparisons of several key parameters of relevant algorithms are conducted to explore the optimum parameters for high accuracy and fast computation speed. A comprehensive set of experiments with existing and new datasets, shows that the method is effective despite pose variations, fast, and appropriate for largescale data, and as accurate as the method with state-of-the-art performance on laboratorybased data. The proposed method was also applied to affective classification of images from the BBC in a task typical for a practical application providing some valuable insights.

show abstract

Representation of facial expression categories in continuous arousal–valence space: Feature and correlation

Cited by 20 publications

References 41 publications

Synchronous prediction of arousal and valence using LSTM network for affective video content analysis

Synchronous prediction of arousal and valence using LSTM network for affective video content analysis

Deep learning-based late fusion of multimodal information for emotion classification of music video

Towards robust automatic affective classification of images using facial expressions for practical applications

Contact Info

Product

Resources

About