Cross-media Structured Common Space for Multimedia Event Extraction

Abstract

We introduce a new task, MultiMedia Event Extraction (M 2 E 2 ), which aims to extract events and their arguments from multimedia documents. We develop the first benchmark and collect a dataset of 245 multimedia news articles with extensively annotated events and arguments. 1 We propose a novel method, Weakly Aligned Structured Embedding (WASE), that encodes structured representations of semantic information from textual and visual data into a common embedding space. The structures are aligned across modalities by employing a weakly supervised training strategy, which enables exploiting available resources without explicit cross-media annotation. Compared to unimodal state-of-the-art methods, our approach achieves 4.0% and 9.8% absolute F-score gains on text event argument role labeling and visual event extraction. Compared to stateof-the-art multimedia unstructured representations, we achieve 8.3% and 5.0% absolute Fscore gains on multimedia event extraction and argument role labeling, respectively. By utilizing images, we extract 21.4% more event mentions than traditional text-only methods.

Cross-Media Keyphrase Prediction: A Unified Framework with Multi-Modality Multi-Head Attention and Image Wordings

“…Cross-media Research. We are also related to cross-media research, where texts and images are jointly exploited for a variety of applications, such as personalized image captioning (Park et al, 2019), event extraction (Li et al, 2020), sarcasm detection (Cai et al, 2019), and text-image relation classification (Vempala and Preotiuc-Pietro, 2019). Some of them have pointed out the usefulness of OCR texts (Chen et al, 2016) and image attributes (Wu et al, 2016) to endow images with higher-level semantics beyond visual features, where we are the first to study how OCR texts and image attributes work together to indicate keyphrases.…”

InfoSurgeon: Cross-Media Fine-grained Information Consistency Checking for Fake News Detection

“…Constructing a KG from each Multimedia News Article: We leverage a publicly available multimedia Information Extraction (IE) system (Li et al, 2020;Lin et al, 2020) to construct a withindocument knowledge graph KG = (N t , E r|a ) for each multimedia article. The IE system can extract 197 types of entities, 61 types of relations, and 144 types of events from text and images.…”

Video Question Answering with Phrases via Semantic Roles

“…SRL in Vision: has been explored in the context of human object interaction (Gupta and Malik, 2015), situation recognition (Yatskar et al, 2016), and multi-media extraction (Li et al, 2020). Most related to ours is the usage of SRLs for grounding (Silberer and Pinkal, 2018) in images and videos (Sadhu et al, 2020).…”