Spatial Knowledge Distillation to Aid Visual Reasoning

Abstract

For tasks involving language and vision, the current state-of-the-art methods tend not to leverage any additional information that might be present to gather relevant (commonsense) knowledge. A representative task is Visual Question Answering where large diagnostic datasets have been proposed to test a system's capability of answering questions about images. The training data is often accompanied by annotations of individual object properties and spatial locations. In this work, we take a step towards integrating this additional privileged information in the form of spatial knowledge to aid in visual reasoning. We propose a framework that combines recent advances in knowledge distillation (teacher-student framework), relational reasoning and probabilistic logical languages to incorporate such knowledge in existing neural networks for the task of Visual Question Answering. Specifically, for a question posed against an image, we use a probabilistic logical language to encode the spatial knowledge and the spatial understanding about the question in the form of a mask that is directly provided to the teacher network. The student network learns from the ground-truth information as well as the teachers prediction via distillation. We also demonstrate the impact of predicting such a mask inside the teachers network using attention. Empirically, we show that both the methods improve the test accuracy over a state-of-the-art approach on a publicly available dataset.

Integrating Knowledge and Reasoning in Image Understanding

“…Appropriateness. Authors in [1,30] used relational reasoning for image question answering. They model relationships as functions between objects and use these functions together to answer questions about an image.…”

“…Several researchers employed commonsense knowledge to enrich high-level understanding tasks such as visual ques- Figure 2: (a) Example of questions that require explicit external knowledge [35], (b) Example where knowledge helps [37]. (c) Ways to integrate background knowledge: i) Pre-process knowledge and augment input [1]; ii) Incorporate knowledge as embeddings [36]; iii) Post-processing using explicit reasoning mechanism [2]; iv) Using knowledge graph to influence NN architecture [24]. tion answering, zero-shot object detection, relationship detection.…”

“…Authors use linguistic knowledge from ConceptNet to predict conditional probabilities (P (pred|subj, obj)) to detect visual relationships from image. [1] uses knowledge distillation to integrate additional supervision about objects to answer questions. Knowledge in Image Retrieval.…”

“…Utilization of background knowledge in data-scarce situations is one of the reasons that necessitate the development of approaches that can utilize such knowledge (from structured or unstructured text) and reason on that knowledge. Additionally, the lack of reasoning and inference capabilities (such as counterfactual, causal queries) of deep learning systems recently started to resurface in various forums [15] 1 . Motivated by these challenges, our goal in this 1 Michael Jordan's views to IEEE Spectrum (bit.ly/ 2GNejBx).…”

“…Additionally, the lack of reasoning and inference capabilities (such as counterfactual, causal queries) of deep learning systems recently started to resurface in various forums [15] 1 . Motivated by these challenges, our goal in this 1 Michael Jordan's views to IEEE Spectrum (bit.ly/ 2GNejBx). Figure 1: The diagram shows the information hierarchy for images and the knowledge associated with each level of information.…”