Visual Sentiment Prediction Based on Automatic Discovery of Affective Regions

Abstract

Abstract-Automatic assessment of sentiment from visual content has gained considerable attention with the increasing tendency of expressing opinions via images and videos online. This paper investigates the problem of visual sentiment analysis, which involves a high-level abstraction in the recognition process. While most of the current methods focus on improving holistic representations, we aim to utilize the local information, which is inspired by the observation that both the whole image and local regions convey significant sentiment information. We propose a framework to leverage affective regions, where we first use an off-the-shelf objectness tool to generate the candidates, and employ a candidate selection method to remove redundant and noisy proposals. Then a convolutional neural network (CNN) is connected with each candidate to compute the sentiment scores, and the affective regions are automatically discovered, taking the objectness score as well as the sentiment score into consideration. Finally, the CNN outputs from local regions are aggregated with the whole images to produce the final predictions. Our framework only requires image-level labels, thereby significantly reducing the annotation burden otherwise required for training. This is especially important for sentiment analysis as sentiment can be abstract, and labeling affective regions is too subjective and labor-consuming. Extensive experiments show that the proposed algorithm outperforms the state-of-the-art approaches on eight popular benchmark datasets.

Smile, Be Happy :) Emoji Embedding for Visual Sentiment Analysis

“…We speculate emoji labeling has the advantage of being universal, finite, and offers an unambiguous one-to-one mapping between label and emo- tion, whereas words carry rich connotations that may make the design of an effective lexicon mapping words to emotions more difficult. Moreover, our SmileyNet outperforms the advanced AR model [42] that employs a customized approach with attention mechanisms when using a single model (K = 1), like ours, and even when using an ensemble of K = 8 models. This is significant given that our model leverages off-the-shelf neural architecture and trained using noisy social media data.…”

“…This further demonstrates the effectiveness of the learned embedding. We hypothesize that our model can be improved even further by employing an ensemble of models like in [42] or customized attention modules such as [13].…”

“…Most of the emotions are predicted with equal accuracy except for anger and fear which show high confusion with the sadness. Finally, similar to [42], we map the 8 emotion classes to positive and negative sentiment and report classification accuracy. Our model outperforms SOTA for this derivative task too, in accordance to our previous results from Sec.…”

Region-based convolutional neural network using group sparse regularization for image sentiment classification

“…Recently, neural network has been widely used [4][5][6][7][8]. Specially, convolutional neural network (CNN)-based sentiment classification methods have shown superior performance of sentiment prediction against traditional label sentiment classification methods for images [6,7].…”

“…Rarely, image sentiment researches based on CNNs consider the recognition of sentiment regions. And these works mainly use the target detection methods to produce the sentimental region candidate set and does not use the local information around the object as a supplement for classification resulting in inaccurate sentiment classification results [4,8]. Moreover, the common regularizations are mainly used for preventing over-fitting in weight level and ignores the sparse character of the network.…”