User Profiling through Deep Multimodal Fusion

Farnadi, Golnoosh; Tang, Jie; Cock, Martine De; Moens, Marie‐Francine

doi:10.1145/3159652.3159691

Cited by 96 publications

(35 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…For example, Tong et al (2017) apply an outer product fusion method to combine text and photo information for the task of detecting human trafficking. For the task of user profiling, formulated as a multi-tasking classification problem, Vijayaraghavan et al (2017) propose a hierarchical attention model; and Farnadi et al (2018) propose the UDMF framework, a hybrid integration model that combines both early feature fusion and later decision fusion using both stacking and power-set combination. Zhong et al (2016) also studied the combination of image and captions for the task of detecting cyberbullying.…”

Section: Introductionmentioning

confidence: 99%

Exploring Deep Multimodal Fusion of Text and Photo for Hate Speech Classification

Yang¹,

Xu²,

Ghosh³

et al. 2019

Proceedings of the Third Workshop on Abusive Language Online

View full text Add to dashboard Cite

Interactions among users on social network platforms are usually positive, constructive and insightful. However, sometimes people also get exposed to objectionable content such as hate speech, bullying, and verbal abuse etc. Most social platforms have explicit policy against hate speech because it creates an environment of intimidation and exclusion, and in some cases may promote real-world violence. As users' interactions on today's social networks involve multiple modalities, such as texts, images and videos, in this paper we explore the challenge of automatically identifying hate speech with deep multimodal technologies, extending previous research which mostly focuses on the text signal alone. We present a number of fusion approaches to integrate text and photo signals. We show that augmenting text with image embedding information immediately leads to a boost in performance, while applying additional attention fusion methods brings further improvement.

show abstract

Section: Introductionmentioning

confidence: 99%

Exploring Deep Multimodal Fusion of Text and Photo for Hate Speech Classification

Yang¹,

Xu²,

Ghosh³

et al. 2019

Proceedings of the Third Workshop on Abusive Language Online

View full text Add to dashboard Cite

show abstract

“…Here we briefly review related works in this direction. There is a large amount of studies on integrating multimodal data sources in deep neural networks, including recommendation [7,11,44], multimodal retrieval [14,23], and user profiling [12], image captioning [10,19]. The flexibility of deep architecture advances the implementation of multimodal fusion either as feature-level fusion or decision-level fusion [31].…”

Section: Deep Multimodal Fusionmentioning

confidence: 99%

Quantifying and Alleviating the Language Prior Problem in Visual Question Answering

Guo

Cheng

Nie

et al. 2019

Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

View full text Add to dashboard Cite

Benefiting from the advancement of computer vision, natural language processing and information retrieval techniques, visual question answering (VQA), which aims to answer questions about an image or a video, has received lots of attentions over the past few years. Although some progress has been achieved so far, several studies have pointed out that current VQA models are heavily affected by the language prior problem, which means they tend to answer questions based on the co-occurrence patterns of question keywords (e.g., how many) and answers (e.g., 2) instead of understanding images and questions. Existing methods attempt to solve this problem by either balancing the biased datasets or forcing models to better understand images. However, only marginal effects and even performance deterioration are observed for the first and second solution, respectively. In addition, another important issue is the lack of measurement to quantitatively measure the extent of the language prior effect, which severely hinders the advancement of related techniques.In this paper, we make contributions to solve the above problems from two perspectives. Firstly, we design a metric to quantitatively measure the language prior effect of VQA models. The proposed metric has been demonstrated to be effective in our empirical studies. Secondly, we propose a regularization method (i.e., score regularization module) to enhance current VQA models by alleviating the language prior problem as well as boosting the backbone model performance. The proposed score regularization module adopts a pair-wise learning strategy, which makes the VQA models answer the question based on the reasoning of the image (upon this question) instead of basing on question-answer patterns observed in the biased training set. The score regularization module is flexible to be integrated into various VQA models. We conducted extensive experiments over two popular VQA datasets (i.e.,

show abstract

“…They use crowdsourcing to annotate user profiles and train log-linear models using lexical features. Farnadi et al [4] merge multiple modalities of user data, such as text, images and relations, to predict age, gender, and personality. They build a hybrid user profiling framework which utilizes a shared representation between modalities to integrate multiple sources of data at feature level and decision level.…”

Section: User Profilingmentioning

confidence: 99%

Using Sentiment Representation Learning to Enhance Gender Classification for User Profiling

Zheng

Zhang

et al. 2019

Web and Big Data

View full text Add to dashboard Cite

User profiling means exploiting the technology of machine learning to predict attributes of users, such as demographic attributes, hobby attributes, preference attributes, etc. It's a powerful data support of precision marketing. Existing methods mainly study network behavior, personal preferences, post texts to build user profile. Through our data analysis of micro-blog, we find that females show more positive and have richer emotions than males in online social platform. This difference is very conducive to the distinction between genders. Therefore, we argue that sentiment context is important as well for user profiling.This paper focuses on exploiting microblog user posts to predict one of the demographic labels: gender. We propose a Sentiment Representation Learning based Multi-Layer Perceptron(SRL-MLP) model to classify gender. First we build a sentiment polarity classifier in advance by training Long Short-Term Memory(LSTM) model on e-commerce review corpus. Next we transfer sentiment representation to a basic MLP network. Last we conduct experiments on gender classification by sentiment representation. Experimental results show that our approach can improve gender classification accuracy by 5.53%, from 84.20% to 89.73%.

show abstract

User Profiling through Deep Multimodal Fusion

Cited by 96 publications

References 24 publications

Exploring Deep Multimodal Fusion of Text and Photo for Hate Speech Classification

Exploring Deep Multimodal Fusion of Text and Photo for Hate Speech Classification

Quantifying and Alleviating the Language Prior Problem in Visual Question Answering

Using Sentiment Representation Learning to Enhance Gender Classification for User Profiling

Contact Info

Product

Resources

About