Cross-Media Keyphrase Prediction: A Unified Framework with Multi-Modality Multi-Head Attention and Image Wordings

Abstract

Social media produces large amounts of contents every day. To help users quickly capture what they need, keyphrase prediction is receiving a growing attention. Nevertheless, most prior efforts focus on text modeling, largely ignoring the rich features embedded in the matching images. In this work, we explore the joint effects of texts and images in predicting the keyphrases for a multimedia post. To better align social media style texts and images, we propose: (1) a novel Multi-Modality Multi-Head Attention (M 3 H-Att) to capture the intricate cross-media interactions; (2) image wordings, in forms of optical characters and image attributes, to bridge the two modalities. Moreover, we design a novel unified framework to leverage the outputs of keyphrase classification and generation and couple their advantages. Extensive experiments on a large-scale dataset 1 newly collected from Twitter show that our model significantly outperforms the previous state of the art based on traditional co-attentions. Further analyses show that our multi-head attention is able to attend information from various aspects and boost classification or generation in diverse scenarios.

Point-of-Interest Type Prediction using Text and Images

“…the best performing EfficientNet model obtains 24.72 F1). This suggests that text encapsulates more relevant information for this task than images on their own, similar to other studies in multimodal computational social science (Wang et al, 2020;Ma et al, 2021).…”

Towards Better Multi-modal Keyphrase Generation via Visual Entity Enhancement and Multi-granularity Image Noise Filtering

“…Compared with the previous models [43,53,54], our model possesses the following two advantages. First, we introduce visual entities that are semantically related to the input image and can be served as anchor points for cross-modal semantic alignment.…”

“…Typically, these studies adopt a coattention network to fuse textual and visual tweet information for recommending hashtags [53,54]. Unlike the studies mentioned above, Wang et al [43] first perform Optical Character Recognition (OCR) to extract explicit optical characters from the input image and then utilize an image captioning model to extract implicit image attributes that reflect the semantic information of the image. To better integrate multi-modal information, they then introduce a multimodal multi-head attention to model the semantic interactions between different modalities.…”

“…In this work, we propose a novel multi-modal keyphrase generation model with visual entity enhancement and image noise filtering. Our model is a significant extension of [43]. Our model not only introduces external visual entities as supplementary information of the textual input, but also leverages multi-granularity noise filtering to effectively exploit the input image.…”

“…In this aspect, the common practice [53,54] use a co-attention network to fuse textual and visual information and then recommend tags for multi-modal tweets. Wang et al [43] propose a multi-modal keyphrase generation model based on an encoder-decoder framework. Typically, the encoder is equipped with a multi-head attention mechanism to fuse multi-modal information, and the decoder is a pointer network.…”