Adapting BERT for Target-Oriented Multimodal Sentiment Classification

Abstract

As an important task in Sentiment Analysis, Target-oriented Sentiment Classification (TSC) aims to identify sentiment polarities over each opinion target in a sentence. However, existing approaches to this task primarily rely on the textual content, but ignoring the other increasingly popular multimodal data sources (e.g., images), which can enhance the robustness of these text-based models. Motivated by this observation and inspired by the recently proposed BERT architecture, we study Target-oriented Multimodal Sentiment Classification (TMSC) and propose a multimodal BERT architecture. To model intra-modality dynamics, we first apply BERT to obtain target-sensitive textual representations. We then borrow the idea from self-attention and design a target attention mechanism to perform target-image matching to derive target-sensitive visual representations. To model inter-modality dynamics, we further propose to stack a set of self-attention layers to capture multimodal interactions. Experimental results show that our model can outperform several highly competitive approaches for TSC and TMSC.

ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations

“…The overall architecture of ZEN is shown in Figure 1, where the backbone model (character encoder) is BERT 4 (Devlin et al, 2018), enhanced by n-gram information represented by a multi-layer encoder. Since the basis of BERT is well explained in previous studies (Devlin et al, 2018;Yu and Jiang, 2019), in this paper, we focus on the details of ZEN, by explaining how n-grams are processed and incorporated into the character encoder.…”

Crossmodal Language Comprehension—Psycholinguistic Insights and Computational Approaches

“…It can be assumed that the use visual information in these systems, to a certain degree, compensates for the lack of genuine language understanding capabilities. Yu and Jiang (2019) reported that using both modalities together is more effective than using them individually with respect to the task of Target-Oriented Sentiment Classification, which determines the sentiment over different individuals, for instance people or places. For this task, the authors propose a neural network that combines the BERT architecture (Devlin et al, 2019) with a target attention mechanism and self-attention layers to model intra-and inter-modality alignments.…”

“…Comparison of using linguistic and visual information individually or together for Target-Oriented Sentiment Classification, evaluated byYu and Jiang (2019) on two publicly available data sets.…”

UI at SemEval-2020 Task 8: Text-Image Fusion for Sentiment Classification

“…In this study, we conducted sentiment classification task on memes by combining the features from images and text, then classified it as positive, negative, or neutral. By using memes as data, it is believed that we can obtained better sentiment classification results because the other data source such as images, can enhance the robustness of models (Yu and Jiang, 2019). The data that used were obtained from SemEval 2020 Task 8: Memotion Analysis (Sharma et al, 2020).…”

“…The results show that the overall performances from combination of the features from text and images are better than only using image or text as features in both datasets. Another study conducted by Yu and Jiang (2019). They proposed a target-oriented multimodal using a pre-trained language representations, BERT architecture, for detecting the sentiment.…”