Multimodal Classification for Analysing Social Media

Abstract

Classification of social media data is an important approach in understanding user behavior on the Web. Although information on social media can be of different modalities such as texts, images, audio or videos, traditional approaches in classification usually leverage only one prominent modality. Techniques that are able to leverage multiple modalities are often complex and susceptible to the absence of some modalities. In this paper, we present simple models that combine information from different modalities to classify social media content and are able to handle the above problems with existing techniques. Our models combine information from different modalities using a pooling layer and an auxiliary learning task is used to learn a common feature space. We demonstrate the performance of our models and their robustness to the missing of some modalities in the emotion classification domain. Our approaches, although being simple, can not only achieve significantly higher accuracies than traditional fusion approaches but also have comparable results when only one modality is available.

ALONE: A Dataset for Toxic Behavior among Adolescents on Twitter

“…On the other hand, although the number of toxic interactions is smaller, they are richer in content as well as multimodal elements, compared to non-toxic interactions [23] (see Tables 5a, 5b, 5c, 5d, and 7). Prior research shows that appropriate incorporation of multimodal elements in modeling with social media data would improve performance [23,24,12,32]. In Table 5a, we see mean and maximum number of tweets per interaction for Toxic ones being significantly higher than Non-toxic ones, suggesting the intensity of the toxic content.…”

“…In Table 5a, we see mean and maximum number of tweets per interaction for Toxic ones being significantly higher than Non-toxic ones, suggesting the intensity of the toxic content. Further, according to Tables 5a, 5b, 5c, 5d, and 7, in the Toxic content, the use of multimodal elements such as image, video, and emoji, is clearly higher, suggesting that the incorporation of these different modalities in the analysis of this dataset will be critical for a reliable outcome [24,23,12,32].…”

Self-Augmented Multi-Modal Feature Embedding

“…There have been a wide variety of neural network-based multi-modal methods and applications. These methods can be roughly categorized into three groups, methods that use network fusion (concatenation) [1,2,3], methods that use gating [4], and methods that use cross-modal training [5]. For time series, it is especially common to combine text with images in document recognition [3,4], natural scene image recognition [1], and cross-modal retrieval [6,7].…”

“…These methods can be roughly categorized into three groups, methods that use network fusion (concatenation) [1,2,3], methods that use gating [4], and methods that use cross-modal training [5]. For time series, it is especially common to combine text with images in document recognition [3,4], natural scene image recognition [1], and cross-modal retrieval [6,7]. Combining audio with video is another common use for multi-modal networks [2,8,9].…”

“…The online handwriting dataset used for the experiments is the Unipen multi-writer isolated digit (1a), uppercase character (1b), and lowercase character (1c) datasets 1 . The datasets were divided into 80% for training and 20% for testing.…”