The New Modality: Emoji Challenges in Prediction, Anticipation, and Retrieval

Cappallo, Spencer; Svetlichnaya, Stacey; Garrigues, P.; Mensink, Thomas; Snoek, Cees G. M.

doi:10.48550/arxiv.1801.10253

Cited by 4 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [11], a model is developed to predict the proper emoji matching a facial expression input. On the other hand, [8] propose to handle emojis as new modality and introduce a model to predict visual or textual concepts by using emo-jis correlations, e.g. learn a ship classifier by leveraging the ship emoji .…”

Section: Related Workmentioning

confidence: 99%

Smile, Be Happy :) Emoji Embedding for Visual Sentiment Analysis

Al-Halah

Aitken

Shi

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

Due to the lack of large-scale datasets, the prevailing approach in visual sentiment analysis is to leverage models trained for object classification in large datasets like Im-ageNet. However, objects are sentiment neutral which hinders the expected gain of transfer learning for such tasks. In this work, we propose to overcome this problem by learning a novel sentiment-aligned image embedding that is better suited for subsequent visual sentiment analysis. Our embedding leverages the intricate relation between emojis and images in large-scale and readily available data from social media. Emojis are language-agnostic, consistent, and carry a clear sentiment signal which make them an excellent proxy to learn a sentiment aligned embedding. Hence, we construct a novel dataset of 4 million images collected from Twitter with their associated emojis. We train a deep neural model for image embedding using emoji prediction task as a proxy. Our evaluation demonstrates that the proposed embedding outperforms the popular object-based counterpart consistently across several sentiment analysis benchmarks. Furthermore, without bell and whistles, our compact, effective and simple embedding outperforms the more elaborate and customized state-of-the-art deep models on these public benchmarks. Additionally, we introduce a novel emoji representation based on their visual emotional response which support a deeper understanding of the emoji modality and their usage on social media.

show abstract

Section: Related Workmentioning

confidence: 99%

Smile, Be Happy :) Emoji Embedding for Visual Sentiment Analysis

Al-Halah

Aitken

Shi

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

View full text Add to dashboard Cite

show abstract

“…text, images, audio, video) (Morency et al, 2011;Poria et al, 2015;Tran and Cambria, 2018). In particular, when we deal with Social Media posts, the presence of both textual and visual content has promoted a number of investigations on sentiment or emotions (Baecchi et al, 2016;You et al, 2016b,a;Yu et al, 2016;Chen et al, 2015) or emojis (Cappallo et al, 2015(Cappallo et al, , 2018.…”

Section: Related Workmentioning

confidence: 99%

Multimodal Emoji Prediction

Barbieri¹,

Ballesteros²,

Ronzano³

et al. 2018

Proceedings of the 2018 Conference of the North American Chapter Of the Association for Computational Linguistics: Hu

View full text Add to dashboard Cite

Emojis are small images that are commonly included in social media text messages. The combination of visual and textual content in the same message builds up a modern way of communication, that automatic systems are not used to deal with. In this paper we extend recent advances in emoji prediction by putting forward a multimodal approach that is able to predict emojis in Instagram posts. Instagram posts are composed of pictures together with texts which sometimes include emojis. We show that these emojis can be predicted by using the text, but also using the picture. Our main finding is that incorporating the two synergistic modalities, in a combined model, improves accuracy in an emoji prediction task. This result demonstrates that these two modalities (text and images) encode different information on the use of emojis and therefore can complement each other.

show abstract

“…Prior work on emoji prediction in text analytics has been done by Francesco et al [3], [4] and emoji prediction in case of images has been done by Cappallo et al [10], [11]. Francesco et al [4] have worked on building models for emoji prediction in case of text messages especially twitter using state-of-theart NLP techniques and also emoji prediction in case of images where they combined both visual and textual features for emoji prediction [3].…”

Section: Related Workmentioning

confidence: 99%

Which Emoji Talks Best for My Picture?

Illendula

Manohar²,

Yedulla³

2018

2018 IEEE/WIC/ACM International Conference on Web Intelligence (WI)

View full text Add to dashboard Cite

Emojis have evolved as complementary sources for expressing emotion in social-media platforms where posts are mostly composed of texts and images. In order to increase the expressiveness of the social media posts, users associate relevant emojis with their posts. Incorporating domain knowledge has improved machine understanding of text. In this paper, we investigate whether domain knowledge for emoji can improve the accuracy of emoji recommendation task in case of multimedia posts composed of image and text. Our emoji recommendation can suggest accurate emojis by exploiting both visual and textual content from social media posts as well as domain knowledge from Emojinet. Experimental results using pre-trained image classifiers and pre-trained word embedding models on Twitter dataset show that our results outperform the current state-ofthe-art by 9.6%. We also present a user study evaluation of our recommendation system on a set of images chosen from MSCOCO dataset.

show abstract

The New Modality: Emoji Challenges in Prediction, Anticipation, and Retrieval

Cited by 4 publications

References 0 publications

Smile, Be Happy :) Emoji Embedding for Visual Sentiment Analysis

Smile, Be Happy :) Emoji Embedding for Visual Sentiment Analysis

Multimodal Emoji Prediction

Which Emoji Talks Best for My Picture?

Contact Info

Product

Resources

About