Discovering Sentimental Interaction via Graph Convolutional Network for Visual Sentiment Prediction

Abstract

With the popularity of online opinion expressing, automatic sentiment analysis of images has gained considerable attention. Most methods focus on effectively extracting the sentimental features of images, such as enhancing local features through saliency detection or instance segmentation tools. However, as a high-level abstraction, the sentiment is difficult to accurately capture with the visual element because of the “affective gap”. Previous works have overlooked the contribution of the interaction among objects to the image sentiment. We aim to utilize interactive characteristics of objects in the sentimental space, inspired by human sentimental principles that each object contributes to the sentiment. To achieve this goal, we propose a framework to leverage the sentimental interaction characteristic based on a Graph Convolutional Network (GCN). We first utilize an off-the-shelf tool to recognize objects and build a graph over them. Visual features represent nodes, and the emotional distances between objects act as edges. Then, we employ GCNs to obtain the interaction features among objects, which are fused with the CNN output of the whole image to predict the final results. Experimental results show that our method exceeds the state-of-the-art algorithm. Demonstrating that the rational use of interaction features can improve performance for sentiment analysis.

Learning to compose diversified prompts for image emotion classification

“…• Based on the object information in the detected image, Ref. [31] built a graph convolutional network based on the sentiment dictionary to explore the relationship between the objects in the image, achieving better performance on the sentiment polarity classification datasets.…”

“…Given the abstract nature of image emotions, it is difficult to obtain sufficient discriminative features from the image itself. However, to improve classification performance, some effort was made to enrich feature representations by incorporating external knowledge, such as proposing a well-designed sentiment dictionary [29][30][31], combining different types of dataset-specific information [32][33][34], or introducing different affective-specific knowledge [8,[35][36][37].…”

“…With the development of deep learning architecture, an increasing number of researchers have managed to capture the internal emotional factors [26][27][28] or used external knowledge [30][31][32][33][34] to improve classification performance, achieving better results than the original deep models. The proposed method, PT-DPC, uses a large-scale pretrained model with richer knowledge, achieving competitive results on both binary and multi-category affective image datasets, obtaining an improvement of 2.9% on the FI 2 dataset and 9.29% on EmotionROI 6 compared to state-of-the-art methods.…”

Develop an Automatic System based on Object Recognition Techniques for Predicting Visual Sentiments from Social Network Images

“…Results showed approximately 85% accuracy for top-level and 79% accuracy for bottom-level emotion tag recognition. A framework that utilizes a Graph Convolutional Network (GCN) was introduced in [18] to extract sentimental interaction features between objects. This framework consists of two branches, one that extracts visual and emotional features from images using a deep network, and the other that extracts emotional interaction features from objects using GCN.…”

Visual Sentiment Analysis from Disaster Images in Social Media

“…Thus, the relation/interaction among different objects in an image is also important to be considered in visual sentiment analysis. In order to incorporate the interactive characteristics of objects, Wu et al [ 30 ] proposed a Graph Convolutional Network (GCN)-based solution to extract interaction features. To this aim, firstly, a CNN model is employed to detect objects in an image.…”