Multimodal Emoji Prediction

Barbieri, Francesco; Ballesteros, Miguel; Ronzano, Francesco; Saggion, Horacio

doi:10.18653/v1/n18-2107

Cited by 45 publications

(18 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[34] recommended emojis should enhance but not replace words, indicating the presence of potential cognitive load. In their task, [35] used pictures along with emojis and found their neural network was able to predict the emoji better when the picture was used versus when the picture was not used. This is a complex factor where two levels of replacement exist as shown in Table 1.…”

Section: Replacementmentioning

confidence: 99%

Pictographs, Ideograms, and Emojis (PIE): A Framework for Empirical Research Using Non-verbal Cues

Suntwal¹,

Brown²,

Brandimarte³

2021

Proceedings of the Annual Hawaii International Conference on System Sciences

View full text Add to dashboard Cite

We propose an empirical framework to understand the impact of non-verbal cues across various research contexts. A large percentage of communication on the Internet uses text-driven non-verbal communication cues often referred to as emojis. Our framework proposes two types of factors to understand the impact of emojis. The first type consists of pictographs, ideograms, and emojis (PIE) factors such as usage, valence, position, and skin tone, and the second type consists of contextual factors depending on the research context, such as fake news, which has high social impact. We discuss how the effect of PIE factors and contextual factors can be used to measure belief, trust, reputation, and intentions across these contexts.

show abstract

Section: Replacementmentioning

confidence: 99%

Pictographs, Ideograms, and Emojis (PIE): A Framework for Empirical Research Using Non-verbal Cues

Suntwal¹,

Brown²,

Brandimarte³

2021

Proceedings of the Annual Hawaii International Conference on System Sciences

View full text Add to dashboard Cite

show abstract

“…The ESLAM first utilizes manually labeled data to train language models and then the noisy labeled data is utilized for smoothing. Another task is to predict the most likely emoji given the text of a tweet [12], [31], [32]. Moreover, emojis have also been employed to detect polarity in tweets [19], [33], [34].…”

Section: Applicationsmentioning

confidence: 99%

Early and Late Fusion of Emojis and Text to Enhance Opinion Mining

Al-Azani

El-Alfy

2021

IEEE Access

View full text Add to dashboard Cite

Opinion mining has gained increasing importance to draw insights from social media content to support decision making. Despite the explosive growth of efforts on linguistic analysis to detect and track people's opinions, more specifically when dealing with aspect-level opinion mining, the results are still away from generalization to real-world applications. Nowadays, emojis are getting excessively popular in social media communication as a complementary way to quickly express opinions and ideas in a visual manner. Two emoji-related issues are highlighted: ambiguity and misinterpretation of emojis' sentiment and tendency of persons to adopt emojis more in positive cases. This paper aims at investigating to what extent the usage of emojis can contribute to the automated detection of sentiment polarity of text messages with focus on Twitter posts in the Arabic language, a widely spoken language but has complex morphology and limited reliable resources for sentiment analysis. For this purpose, after an extensive review of the stateof-the-art of emojis-related work, a dataset is composed and several feature extraction methods are applied for both text and emojis modalities. Moreover, various early and late fusion techniques are proposed to combine both modalities at different levels including feature, score, decision and hybrid. The experimental results revealed that emojis features can significantly improve the classification results, especially when integrated with text at the score level.

show abstract

“…Most of the previous works emphasize the use of emojis instead of stickers. For example, [5,6] use a multimodal approach to recommend emojis based on the text and images in an Instagram post. However, emojis are typically used in conjunction with text, while stickers are independent information carriers.…”

Section: Related Workmentioning

confidence: 99%

Learning to Respond with Stickers: A Framework of Unifying Multi-Modality in Multi-Turn Dialog

Gao

Chen

Liu

et al. 2020

Proceedings of the Web Conference 2020

View full text Add to dashboard Cite

Stickers with vivid and engaging expressions are becoming increasingly popular in online messaging apps, and some works are dedicated to automatically select sticker response by matching text labels of stickers with previous utterances. However, due to their large quantities, it is impractical to require text labels for the all stickers. Hence, in this paper, we propose to recommend an appropriate sticker to user based on multi-turn dialog context history without any external labels. Two main challenges are confronted in this task. One is to learn semantic meaning of stickers without corresponding text labels. Another challenge is to jointly model the candidate sticker with the multi-turn dialog context. To tackle these challenges, we propose a sticker response selector (SRS) model. Specifically, SRS first employs a convolutional based sticker image encoder and a self-attention based multi-turn dialog encoder to obtain the representation of stickers and utterances. Next, deep interaction network is proposed to conduct deep matching between the sticker with each utterance in the dialog history. SRS then learns the short-term and long-term dependency between all interaction results by a fusion network to output the the final matching score. To evaluate our proposed method, we collect a large-scale realworld dialog dataset with stickers from one of the most popular online chatting platform. Extensive experiments conducted on this dataset show that our model achieves the state-of-the-art performance for all commonly-used metrics. Experiments also verify the * Equal contribution. Ordering is decided by a coin flip. Work performed during an internship at IIAI. † WICT is the abbreviation of Wangxuan Institute of Computer Technology. effectiveness of each component of SRS. To facilitate further research in sticker selection field, we release this dataset of 340K multi-turn dialog and sticker pairs 1 .

show abstract

Multimodal Emoji Prediction

Cited by 45 publications

References 32 publications

Pictographs, Ideograms, and Emojis (PIE): A Framework for Empirical Research Using Non-verbal Cues

Pictographs, Ideograms, and Emojis (PIE): A Framework for Empirical Research Using Non-verbal Cues

Early and Late Fusion of Emojis and Text to Enhance Opinion Mining

Learning to Respond with Stickers: A Framework of Unifying Multi-Modality in Multi-Turn Dialog

Contact Info

Product

Resources

About