Graph-Structured Representations for Visual Question Answering

Teney, Damien; Liu, Lingqiao; Hengel, Anton van den

doi:10.1109/cvpr.2017.344

Cited by 423 publications

(233 citation statements)

References 26 publications

Supporting

Mentioning

233

Contrasting

Order By: Relevance

“…[6] extends RNs with the Broadcasting Convolutional Network module, which globally broadcasts objects' visuo-spatial features. The first work to use graph networks in VQA is [34], which combines dependency parses of questions and scene graph representations of abstract scenes. [45] proposes modeling structured visual attention over a Conditional Random Field on image regions.…”

Section: Graph Network and Contextualized Representationsmentioning

confidence: 99%

Language-Conditioned Graph Networks for Relational Reasoning

Rohrbach

Darrell

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

156

View full text Add to dashboard Cite

Solving grounded language tasks often requires reasoning about relationships between objects in the context of a given task. For example, to answer the question "What color is the mug on the plate?" we must check the color of the specific mug that satisfies the "on" relationship with respect to the plate. Recent work has proposed various methods capable of complex relational reasoning. However, most of their power is in the inference structure, while the scene is represented with simple local appearance features. In this paper, we take an alternate approach and build contextualized representations for objects in a visual scene to support relational reasoning. We propose a general framework of Language-Conditioned Graph Networks (LCGN), where each node represents an object, and is described by a context-aware representation from related objects through iterative message passing conditioned on the textual input. E.g., conditioning on the "on" relationship to the plate, the object "mug" gathers messages from the object "plate" to update its representation to "mug on the plate", which can be easily consumed by a simple classifier for answer prediction. We experimentally show that our LCGN approach effectively supports relational reasoning and improves performance across several tasks and datasets.

show abstract

Section: Graph Network and Contextualized Representationsmentioning

confidence: 99%

Language-Conditioned Graph Networks for Relational Reasoning

Rohrbach

Darrell

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

156

View full text Add to dashboard Cite

show abstract

“…Another promising attempt is to exploit graph-structured representations in VQA [128], [129], where object relations and language structures are represented as graphs whose structure information can be further explored via techniques such as graph convolutional networks (GCN). As is shown in Figure 23, Norcliffe-Brown et al [129] propose a graph-based approach for visual question answering.…”

Section: B Exemplar Applications Of Data and Knowledge Fusionmentioning

confidence: 99%

Multi-Modal Deep Analysis for Multimedia

Zhu

Wang

Zhang

2020

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

With the rapid development of Internet and multimedia services in the past decade, a huge amount of usergenerated and service provider-generated multimedia data become available. These data are heterogeneous and multi-modal in nature, imposing great challenges for processing and analyzing them. Multi-modal data consist of a mixture of various types of data from different modalities such as texts, images, videos, audios etc. In this article, we present a deep and comprehensive overview for multi-modal analysis in multimedia. We introduce two scientific research problems, data-driven correlational representation and knowledge-guided fusion for multimedia analysis. To address the two scientific problems, we investigate them from the following aspects: 1) multi-modal correlational representation: multi-modal fusion of data across different modalities, and 2)multi-modal data and knowledge fusion: multi-modal fusion of data with domain knowledge. More specifically, on datadriven correlational representation, we highlight three important categories of methods, such as multi-modal deep representation, multi-modal transfer learning, and multi-modal hashing. On knowledge-guided fusion, we discuss the approaches for fusing knowledge with data and four exemplar applications that require various kinds of domain knowledge, including multi-modal visual question answering, multi-modal video summarization, multimodal visual pattern mining and multi-modal recommendation. Finally, we bring forward our insights and future research directions.

show abstract

“…These graph neural networks have been widely employed in various tasks of computer vision and have made very promising progress, e.g. object parsing [31,32], multi-label image recognition [52], visual question answer [46], social relationship understanding [51], person re-identification [42] and action recognition [49]. These work create knowledge graph based on the relationship of different entities, e.g.…”

Section: Datasetsmentioning

confidence: 99%

Fashion Retrieval via Graph Reasoning Networks on a Similarity Pyramid

Kuang¹,

Gao

et al. 2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Matching clothing images from customers and online shopping stores has rich applications in E-commerce. Existing algorithms encoded an image as a global feature vector and performed retrieval with the global representation. However, discriminative local information on clothes are submerged in this global representation, resulting in suboptimal performance. To address this issue, we propose a novel Graph Reasoning Network (GRNet) on a Similarity Pyramid, which learns similarities between a query and a gallery cloth by using both global and local representations in multiple scales. The similarity pyramid is represented by a Graph of similarity, where nodes represent similarities between clothing components at different scales, and the final matching score is obtained by message passing along edges. In GRNet, graph reasoning is solved by training a graph convolutional network, enabling to align salient clothing components to improve clothing retrieval. To facilitate future researches, we introduce a new benchmark FindFashion, containing rich annotations of bounding boxes, views, occlusions, and cropping. Extensive experiments show that GRNet obtains new state-of-the-art results on two challenging benchmarks, e.g. pushing the top-1, top-20, and top-50 accuracies on DeepFashion to 26%, 64%, and 75% (i.e. 4%, 10%, and 10% absolute improvements), outperforming competitors with large margins. On FindFashion, GRNet achieves considerable improvements on all empirical settings.

show abstract

Graph-Structured Representations for Visual Question Answering

Cited by 423 publications

References 26 publications

Language-Conditioned Graph Networks for Relational Reasoning

Language-Conditioned Graph Networks for Relational Reasoning

Multi-Modal Deep Analysis for Multimedia

Fashion Retrieval via Graph Reasoning Networks on a Similarity Pyramid

Contact Info

Product

Resources

About