Task-agnostic representation learning of multimodal twitter data for downstream applications

Abstract

Twitter is a frequent target for machine learning research and applications. Many problems, such as sentiment analysis, image tagging, and location prediction have been studied on Twitter data. Much of the prior work that addresses these problems within the context of Twitter focuses on a subset of the types of data available, e.g. only text, or text and image. However, a tweet can have several additional components, such as the location and the author, that can also provide useful information for machine learning tasks. In this work, we explore the problem of jointly modeling several tweet components in a common embedding space via task-agnostic representation learning, which can then be used to tackle various machine learning applications. To address this problem, we propose a deep neural network framework that combines text, image, and graph representations to learn joint embeddings for 5 tweet components: body, hashtags, images, user, and location. In our experiments, we use a large dataset of tweets to learn a joint embedding model and use it in multiple tasks to evaluate its performance vs. state-of-the-art baselines specific to each task. Our results show that our proposed generic method has similar or superior performance to specialized application-specific approaches, including accuracy of 52.43% vs. 48.88% for location prediction and recall of up to 15.93% vs. 12.12% for hashtag recommendation.

Cross-modality representation learning from transformer for hashtag prediction

“…Then, it merges the image and text representations and feeds the output values into GRU networks to generate a sequence of the recommended hashtags (decoder). • Tweet Embedding Network: TweetEmbd_Net [34] proposed a task-agnostic model for multimodal Twitter data. This model combines the tweet components (image, text, hashtags, user, location, and time) for representation learning for downstream applications, including hashtag recommendation tasks.…”

“…Connection Network for multimodal feature fusion. The work [34] proposed a deep neural network framework that combines the tweet components for representation learning of multimodal Twitter data for downstream applications, including hashtag recommendations tasks. An overview of the selected literature is shown in Table 1.…”

A Systematic Literature Review on Multimodal Machine Learning: Applications, Challenges, Gaps and Future Directions

“…Of 8 papers in the geoscience domain, 5 were in classification [347]- [351], 1 was in recognition [352], detection [353], and precision [354]. In total 10 articles were identified in the social media domain, where 4 were in detection [355]- [358], 2 were in analysis [359], [360] and 1 was in classification [361], prediction [362], recommendation [363] and verification [364]. In the vehicle domain, 6 articles were found; from them, 2 were in recognition [365], [366], 1 was in identification [367] and 3 were in detection [368]- [370].…”

“…[95], [116], [168], [229], [244], [273], [274], [289], [300], [336], [355], [362] articles, respectively. In total, 7 and 5 articles related to similarity [48], [51], [125], [170], [171], [268], [338] and structured [64], [199], [211], [212], [262]…”

“…Graphical models frame representation using implicit random variables where probabilistic graphical models such as deep Boltzmann machines (DBM) [113] are used. The sequential representation uses sequential models where hidden states are considered to represent the data, such as recurrent neural networks (RNNs) [289], [355], [362]. From three subtypes of joint representation, NNs were used primarily because of their superior performance, but they could not handle missing data.…”