Reasoning over Vision and Language: Exploring the Benefits of Supplemental Knowledge

Abstract

The limits of applicability of vision-andlanguage models are defined by the coverage of their training data. Tasks like vision question answering (VQA) often require commonsense and factual information beyond what can be learned from task-specific datasets. This paper investigates the injection of knowledge from general-purpose knowledge bases (KBs) into vision-and-language transformers. We use an auxiliary training objective that encourages the learned representations to align with graph embeddings of matching entities in a KB. We empirically study the relevance of various KBs to multiple tasks and benchmarks. The technique brings clear benefits to knowledge-demanding question answering tasks (OK-VQA, FVQA) by capturing semantic and relational knowledge absent from existing models. More surprisingly, the technique also benefits visual reasoning tasks (NLVR2, SNLI-VE). We perform probing experiments and show that the injection of additional knowledge regularizes the space of embeddings, which improves the representation of lexical and semantic similarities. The technique is model-agnostic and can expand the applicability of any vision-and-language transformer with minimal computational overhead.

A Thousand Words Are Worth More Than a Picture: Natural Language-Centric Outside-Knowledge Visual Question Answering

“…More recently, the dataset outsideknowledge visual question answering (OK-VQA) [39] is proposed where the usage of outside knowledge is open to the entire web. Most existing work for OK-VQA rely on the pre-trained vision-language models as a major workhorse for question answering [12,36,38,48,60,63]. In [12,48], learned knowledge embeddings are injected into visionlanguage models to perform knowledge-aware question answering.…”

“…Most existing work for OK-VQA rely on the pre-trained vision-language models as a major workhorse for question answering [12,36,38,48,60,63]. In [12,48], learned knowledge embeddings are injected into visionlanguage models to perform knowledge-aware question answering. Other work uses vision-language models as a knowledge-free VQA model first and later adjusts the predicted answers by fusion with knowledge graphs [38] or answer validation with knowledge text [60].…”

“…First, most previous methods utilize the vision-language model as the backbone for question answering and then integrate it with external knowledge. Some represent the knowledge in the form of graph (KRISP [38], Concept-Bert [12], RVLESK [48]) while others fuse the output of the vision-language model with textual knowledge representation (MAVEx [60]) or implicit knowledge from a language QA mdoel [46]. Second, a concurrent work, VRR [36], transforms the image into caption text and performs spanbased question answering on a trimmed knowledge base using Google search engine.…”

“…Second, a concurrent work, VRR [36], transforms the image into caption text and performs spanbased question answering on a trimmed knowledge base using Google search engine. Last and most importantly, all of the above methods achieve very similar VQA scores between 38.60 and 39.4, despite usage of diverse sources of knowledge bases such as ConceptNet [12,38,48,60], Google Image [60], Google Web Search [36] and Wikipiedia [60] and pretraining on other datasets such as VQA [12,38,48,60] and Visual Genome [48].…”

“…One solution is to first fuse the image and question information in the multi-modal space with pre-trained vision-language models, and then inject knowledge into the multi-modal space. Most previous work on OK-VQA follow this paradigm, including directly injecting the knowledge embeddings [12,48] and fusing the output of a vision-language model with the knowledge graph through graph convolutional network [38]. However, this paradigm is at the cost of squeezing the rich representation of the textual knowledge, in the magnitude of hundreds of millions, into a much smaller multi-modal space.…”