Weakly Supervised Coupled Networks for Visual Sentiment Analysis

Abstract

Automatic assessment of sentiment from visual content has gained considerable attention with the increasing tendency of expressing opinions on-line. In this paper, we solve the problem of visual sentiment analysis using the high-level abstraction in the recognition process. Existing methods based on convolutional neural networks learn sentiment representations from the holistic image appearance. However, different image regions can have a different influence on the intended expression. This paper presents a weakly supervised coupled convolutional network with two branches to leverage the localized information. The first branch detects a sentiment specific soft map by training a fully convolutional network with the cross spatial pooling strategy, which only requires image-level labels, thereby significantly reducing the annotation burden. The second branch utilizes both the holistic and localized information by coupling the sentiment map with deep features for robust classification. We integrate the sentiment detection and classification branches into a unified deep framework and optimize the network in an end-to-end manner. Extensive experiments on six benchmark datasets demonstrate that the proposed method performs favorably against the state-ofthe-art methods for visual sentiment analysis.

WSCNet: Weakly Supervised Coupled Networks for Visual Sentiment Classification and Detection

“…A preliminary version of this work appeared in CVPR [1]. Examples from the (a) EmotionROI [14] and (b) EMOd datasets [15] with the human annotation.…”

“…This paper is an extended version of our conference paper [1], to which we enrich the contributions in the following four aspects: (1) We provide useful details of our weakly supervised framework, and distinguish it from comparative methods, e.g., salience detection and weakly supervised detection frameworks. (2) We add a comprehensive review of related work making the manuscript more self-contained.…”

Affective Computing for Large-scale Heterogeneous Multimedia Data

“…Extending triplet constraints to a hierarchical structure, the sentiment constraint employs a sentiment vector based on the texture information from the convolutional layer to measure the difference between affective images. In [105,139], Yang et al proposed a weakley supervised coupled convolutional neural network to exploit the discriminability of localized regions for emotion classification. Based on the image-level labels, a sentiment map is firstly detected in one branch with the cross spatial pooling strategy.…”

CycleEmotionGAN: Emotional Semantic Consistency Preserved CycleGAN for Adapting Image Emotions

“…Yang et al (2018b) employed deep metric learning to optimize both the retrieval and classification tasks by jointly optimizing cross-entropy loss and a novel sentiment constraint. Different from improving global image representations, several methods (You, Jin, and Luo 2017;Yang et al 2018a) consider the local information for IER.…”

“…To tackle the subjectivity issue, we can predict personalized emotion perceptions for each viewer (Zhao et al 2016), or learn the emotion distributions for each image (Yang, She, and Sun 2017;Zhao et al 2017a;. With the advent of deep neural networks, several end-toend approaches have been proposed to classify image emotions (Rao, Xu, and Xu 2016;You et al 2016;Zhu et al 2017b;Yang et al 2018a) or learn emotion distributions (Peng et al 2015;Yang, She, and Sun 2017). Current IER methods, especially ones based on deep neural networks, perform well with large-scale labelled training data.…”