Attentive Relational Networks for Mapping Images to Scene Graphs

Abstract

Scene graph generation refers to the task of automatically mapping an image into a semantic structural graph, which requires correctly labeling each extracted object and their interaction relationships. Despite the recent success in object detection using deep learning techniques, inferring complex contextual relationships and structured graph representations from visual data remains a challenging topic. In this study, we propose a novel Attentive Relational Network that consists of two key modules with an object detection backbone to approach this problem. The first module is a semantic transformation module utilized to capture semantic embedded relation features, by translating visual features and linguistic features into a common semantic space. The other module is a graph self-attention module introduced to embed a joint graph representation through assigning various importance weights to neighboring nodes. Finally, accurate scene graphs are produced by the relation inference module to recognize all entities and the corresponding relations. We evaluate our proposed method on the widely-adopted Visual Genome Dataset, and the results demonstrate the effectiveness and superiority of our model.

Counterfactual Critic Multi-Agent Training for Scene Graph Generation

“…or tree-near-?. Such intuition has been empirically shown benefits in boosting SGG [62,7,28,30,29,71,20,73,58,13,44,59,45]. More specifically, these methods use a conditional random field [79] to model the joint distribution of nodes and edges, where the context is incorporated by message passing among the nodes through edges via a multi-step meanfield approximation [26]; then, the model is optimized by the sum of cross-entropy (XE) losses of nodes (e.g., objects) and edges (e.g., relationships).…”

Attentive Gated Graph Neural Network for Image Scene Graph Generation

“…Ref. [29] proposed an attentive relational network, which use self-attention mechanism to sparse the connections in the graph. However, the built graph is static and cannot simulate the process of message propagation.…”

“…Implementation Details We implement our model based on TensorFlow [33] framework on the NVIDIA 2080 Ti GPU. Similar to prior works for scene graph generation [13,28,29], we adopt Faster R-CNN detector (with VGG16 pretrained in ImageNet dataset) [1] as backbone in feature extraction module. During training, the number of proposals from RPN is 256.…”

“…Evaluation Metrics. Following [28,29], Top-K Recall (denoted as Rec@K ) is used to evaluate how many labelled relationships are hit in the Top-K predictions. The reason we use Recall instead of mean Average Precision (mAP) is that annotations of the relationships are not complete.…”

Triplet-Aware Scene Graph Embeddings

“…Generating scene graphs from visual features [22,24,23,4,3,14,26,20,13,5] is a relatively explored task. Wan et al [19] specifically predicts new triplets for scene graph completion using existing scene graphs and visual features.…”