Multimodal Named Entity Disambiguation for Noisy Social Media Posts

Abstract

We introduce the new Multimodal Named Entity Disambiguation (MNED) task for multimodal social media posts such as Snapchat or Instagram captions, which are composed of short captions with accompanying images. Social media posts bring significant challenges for disambiguation tasks because 1) ambiguity not only comes from polysemous entities, but also from inconsistent or incomplete notations, 2) very limited context is provided with surrounding words, and 3) there are many emerging entities often unseen during training. To this end, we build a new dataset called SnapCaptionsKB, a collection of Snapchat image captions submitted to public and crowd-sourced stories, with named entity mentions fully annotated and linked to entities in an external knowledge base. We then build a deep zeroshot multimodal network for MNED that 1) extracts contexts from both text and image, and 2) predicts correct entity in the knowledge graph embeddings space, allowing for zeroshot disambiguation of entities unseen in training set as well. The proposed model significantly outperforms the stateof-the-art text-only NED models, showing efficacy and potentials of the MNED task.

Multimodal Entity Linking for Tweets

“…To our knowledge, [36] is the only other work that leverages multimodal information for an entity disambiguation task in a social media context. However, they use a dataset of 12K annotated image-caption pairs from Snapchat (which is not made available), whereas our work relies on a much larger base (85K samples for the benchmark and 2M to build the KB) and is more in line with the state-of-the-art which mostly uses Twitter as case study for EL on social media.…”

“…In our case, due to the nature of the dataset, we only use a simple inclusion (i.e. the mention words are present in the entity name) but a more complex lexical distance that takes into account more variations could be considered [36]. The disambiguation of m i is then formalized as finding the best multimodal similarity measure between the tweet containing the mention and the timeline of the correct entity, both containing text and images:…”

“…Since our approach is not directly comparable with previous works [36], we compare the results with different configurations and baselines. As 50% of mentions are unique in each split of our dataset, we do not take the standard most frequent entity prediction as a baseline, which links the entity that was seen most in training to its corresponding mention.…”

Chinese Short Text Entity Disambiguation Based on the Dual-Channel Hybrid Network

“…The data set is obtained from the entity linking task for Chinese short text from the China Conference on Knowledge Graph and Semantic Computing (CCKS 2019) 1 . There are 90,000 pieces of data in the data set and 399,252 pieces of data in the external knowledge base, all of which are short Chinese texts.…”

“…Entity disambiguation is an important task in knowledge graphs and a key technology in the field of information extraction and integration that affects the accuracy of many downstream tasks in natural language processing, such as information retrieval and intelligent question answering. Recently, the research on entity disambiguation based on knowledge graphs has gradually increased, for example, Seungwhan Moon et al [1], Prabal Agarwal et al [2]. Causes of entity ambiguity can be classified into diversity and ambiguity, namely, synonymy and polysemy [3].…”