Visual Query Answering by Entity-Attribute Graph Matching and Reasoning

Abstract

Visual Query Answering (VQA) is of great significance in offering people convenience: one can raise a question for details of objects, or high-level understanding about the scene, over an image. This paper proposes a novel method to address the VQA problem. In contrast to prior works, our method that targets single scene VQA, replies on graphbased techniques and involves reasoning. In a nutshell, our approach is centered on three graphs. The first graph, referred to as inference graph G I , is constructed via learning over labeled data. The other two graphs, referred to as query graph Q and entity-attribute graph G EA , are generated from natural language query Q nl and image Img, that are issued from users, respectively. As G EA often does not take sufficient information to answer Q, we develop techniques to infer missing information of G EA with G I . Based on G EA and Q, we provide techniques to find matches of Q in G EA , as the answer of Q nl in Img. Unlike commonly used VQA methods that are based on end-to-end neural networks, our graph-based method shows well-designed reasoning capability, and thus is highly interpretable. We also create a dataset on soccer match (Soccer-VQA) with rich annotations. The experimental results show that our approach outperforms the state-of-the-art method and has high potential for future investigation.

Deep Metric Learning with Graph Consistency

“…IRG (Liu et al 2019) employs the graph relation for knowledge distillation. The graph knowledge is also used for visual query answering (Xiong et al 2019). However, different from these works, we aim at encouraging the discrimination of the learned deep embedding by regularizing the randomly constructed sub-graphs over data points to be consistent with each other, which is the obvious property of our defined 'Discriminative Graph' and discriminative feature distribution.…”

Temporal Reasoning Graph for Activity Recognition

“…In the face clustering problem, several works attempted to modeling pair-wise relationships for face graph generation [45], [49]. To reason between different instances in visual question answering tasks, Xiong et al [47] proposed a graph matching module for investigating such relation. In sketch-based action recognition, several works showed that modeling interactions of sketch joint can achieve excellent performance [25], [26], [48].…”

“…Reasoning about the relation between instance over time in the video is critical for activity recognition [53]. In addition, modeling relations between vision objects have become a popular problem in computer vision [18], [45], [47], [51]. The most straightforward visual reasoning task is object relation reasoning in CLEVR benchmark [13], and significant efforts have been devoted to a variety of traditional visual tasks with pairwise relationship reasoning.…”

Knowledge-Based Video Question Answering with Unsupervised Scene Descriptions

“…Most scene graph methods consist on an object detector, an attribute classifier and a relationship predictor [48,19,54,50,55,56]. Scene graphs have been used in multiple vision and language tasks, including image captioning [51,7,54] and VQA [36,32,46]. However, less attention has been paid to generating scene graphs from videos, in which relationships are both spatial and temporal.…”