Computational construction grammar for visual question answering

Abstract

In order to be able to answer a natural language question, a computational system needs three main capabilities. First, the system needs to be able to analyze the question into a structured query, revealing its component parts and how these are combined. Second, it needs to have access to relevant knowledge sources, such as databases, texts or images. Third, it needs to be able to execute the query on these knowledge sources. This paper focuses on the first capability, presenting a novel approach to semantically parsing questions expressed in natural language. The method makes use of a computational construction grammar model for mapping questions onto their executable semantic representations. We demonstrate and evaluate the methodology on the CLEVR visual question answering benchmark task. Our system achieves a 100% accuracy, effectively solving the language understanding part of the benchmark task. Additionally, we demonstrate how this solution can be embedded in a full visual question answering system, in which a question is answered by executing its semantic representation on an image. The main advantages of the approach include (i) its transparent and interpretable properties, (ii) its extensibility, and (iii) the fact that the method does not rely on any annotated training data.

Constructionist approaches: Past, present, future

“…answering text questions about images; Nevens et al 2019) and a platform for analyzing opinions on social media (Willaert et al 2020).…”

The FCG Editor: An innovative environment for engineering computational construction grammars

“…On the theoretical side, they are a crucial instrument supporting the assessment of the consistency and coverage of construction grammar analyses. On the practical side, they facilitate the use of construction grammar insights and analyses in language technology applications, such as visual question answering systems [ 30 ], the frame-semantic analysis of discourse [ 31 , 32 ] and tools for exploring large corpora from a construction grammar perspective [ 46 ].…”

“…The substantive body of research that has by now been yielded by the field of computational construction grammar has not only helped to establish more solid foundations for the constructionist view on language, but has in the meantime also resulted in a number of impactful real-world applications [ 30 – 32 , 46 , 58 – 61 ]. Yet, the grammars that are currently available are either fragments targeted towards detailed analyses of specific linguistic phenomena of interest, including the English auxiliary system [ 62 , 63 ], English measure phrases [ 64 ], English caused-motion constructions [ 23 , 65 ], English long-distance dependencies, [ 42 , 66 ], English metaphors [ 67 ], Dutch modal stacking [ 68 ], Hungarian poly-personal agreement [ 69 ] and tense, aspect and modality in the Spanish verbal system [ 70 ], or application-specific grammars that were designed for optimal performance on a predefined task [ 30 , 32 ]. Some attempts have been made to create large, domain-general, fine-grained computational construction grammars, either by leveraging FrameNet data to expand the coverage of seed grammars [ 71 , 72 ], or by combining a set of fully instantiated constructions that were automatically created based on lexical resources with a collection of hand-crafted constructions that handle more abstract patterns [ 73 ].…”

“…Babel is marketed as an all-in-one software library for setting up multi-agent experiments on the emergence and evolution of communication and language [ 104 ]. Babel provides building blocks for implementing all aspects involved in such experiments, ranging from the sensory-motor level [ 105 ], through the conceptual level [ 106 ], to the linguistic level [ 36 ]. In essence, Babel is not a computer program or editor, but a software library that includes FCG as a component.…”

Neuro-Symbolic Procedural Semantics for Reasoning-Intensive Visual Dialogue Tasks

“…In this paper, we introduce the use of neuro-symbolic procedural semantic representations for solving visual dialogue tasks. We build further on earlier work in the area of visual question answering, in which procedural semantic representations have already been successfully used for representing the meaning of questions in the form of executable queries [2,16,26]. Such procedural semantic representations capture the logical structure underlying a question, and can be directly executed on a given image to compute an answer.…”