Weakly-Supervised Visual-Retriever-Reader for Knowledge-based Question Answering

Abstract

Knowledge-based visual question answering (VQA) requires answering questions with external knowledge in addition to the content of images. One dataset that is mostly used in evaluating knowledge-based VQA is OK-VQA, but it lacks a gold standard knowledge corpus for retrieval. Existing work leverage different knowledge bases (e.g., ConceptNet and Wikipedia) to obtain external knowledge. Because of varying knowledge bases, it is hard to fairly compare models' performance. To address this issue, we collect a natural language knowledge base that can be used for any VQA system. Moreover, we propose a Visual Retriever-Reader pipeline to approach knowledge-based VQA. The visual retriever aims to retrieve relevant knowledge, and the visual reader seeks to predict answers based on given knowledge. We introduce various ways to retrieve knowledge using text and images and two reader styles: classification and extraction. Both the retriever and reader are trained with weak supervision. Our experimental results show that a good retriever can significantly improve the reader's performance on the OK-VQA challenge. The code and corpus are provided in this link. * Equal contributionQuestion: What sort of vehicle used this item? Answer: fire truck LXMERT: truck LXMERT + Caption: fire truck Ours: fire truck kn: fire engine, also called fire truck, mobile (nowadays selfpropelled) piece of equipment used in firefighting.... Caption: a red fire hydrant sitting on the side of a road. Question: Where did this sport originate? Answer: australia, hawaii, polynesian LXMERT: california LXMERT + Caption: california Ours: hawaii kn: surfing was invented in hawaii... Caption: a man riding a wave on a surfboard in the ocean.

KAT: A Knowledge Augmented Transformer for Vision-and-Language

“…ViLBERT (Lu et al, 2019) and LXMERT (Tan and Bansal, 2019) propose a two-stream architecture to process images and text independently and fused by a third transformer in ta later stage. While these models have shown to store in-depth cross-modal knowledge and achieved impressive performance on knowledge-based VQA (Marino et al, 2021;Wu et al, 2022;Luo et al, 2021), this type of implicitly learned knowledge is not sufficient to answer many knowledge-based questions (Marino et al, 2021). Another line of work for multimodal transformers, such as CLIP (Radford et al, 2021) or ALIGN (Jia et al, 2021), aligns visual and language representations by contrastive learning.…”

“…Recent approaches have shown a great potential to incorporate external knowledge for knowledgebased VQA. Several methods explore aggregating the external knowledge either in the form of structured knowledge graphs (Garderes et al, 2020;Narasimhan et al, 2018;Li et al, 2020b;Wang et al, 2017a,b), unstructured knowledge bases (Marino et al, 2021;Wu et al, 2022;Luo et al, 2021), and neural-symbolic inference based knowledge (Chen et al, 2020;West et al, 2021). In these methods, object detectors (Ren et al, 2015) and scene classifiers (He et al, 2016) are used to associate images with external knowledge.…”

“…In these methods, object detectors (Ren et al, 2015) and scene classifiers (He et al, 2016) are used to associate images with external knowledge. Further, external APIs, such as Google (Wu et al, 2022;Luo et al, 2021), Microsoft (Chen et al, 2021a;Yang et al, 2022) and OCR (Luo et al, 2021;Wu et al, 2022) are used to enrich the associated knowledge. Finally, pre-trained transformerbased language models (Chen et al, 2021a;Yang et al, 2022), or multimodal models (Wu et al, 2022;Luo et al, 2021;Wu et al, 2022;Garderes et al, 2020;Marino et al, 2021) are leveraged as implicit knowledge bases for answer predictions.…”

“…While part of our approach is similar to PICa (Yang et al, 2022) which considers GPT-3 as implicit knowledge base, our model takes one step further by showing that how explicit and implicit knowledge can be integrated during knowledge reasoning. Another similar work Vis-DPR (Luo et al, 2021) collects a knowledge corpus from training set by Google Search which is specific to a certain dataset. Our proposed model is more generic by collecting entities from Wikidata and not limited to the training set.…”

Neural Retriever and Go Beyond: A Thesis Proposal

“…Result We evaluate three retrievers on OK-VQA dataset and use the knowledge base (with 112,724 pieces of knowledge) created in (Luo et al, 2021b) as the corpus. We retrieve 1/5/10/20/50/80/100 knowledge for each question.…”