Semantic Frames and Visual Scenes: Learning Semantic Role Inventories from Image and Video Descriptions

Abstract

Frame-semantic parsing and semantic role labelling, that aim to automatically assign semantic roles to arguments of verbs in a sentence, have recently become an active strand of research in NLP. However, to date these methods have relied on a predefined inventory of semantic roles. In this paper, we present a method to automatically learn argument role inventories for verbs from large corpora of text, images and videos. We evaluate the method against manually constructed role inventories in FrameNet and show that the visual model outperforms the language-only model and operates with a high precision.

Multimodal Grounding for Language Processing

“…In open-domain language understanding, semantic disambiguation is even more challenging. Approaches using multimodal information for the disambiguation of concepts (Xie et al, 2017), named entities (Moon et al, 2018), and sentences (Botschen et al, 2018;Shutova et al, 2017) show promising tendencies, but the underlying compositional principles are not yet understood.…”