Towards Context-Aware Interaction Recognition for Visual Relationship Detection

Zhuang, Bohan; Liu, Lingqiao; Shen, Chunhua; Reid, Ian

doi:10.1109/iccv.2017.71

Cited by 155 publications

(121 citation statements)

References 39 publications

(110 reference statements)

Supporting

Mentioning

121

Contrasting

Order By: Relevance

“…A visual relationship is discovered (starting from the proposals coming from an object detector) with a graph traversal algorithm in a reinforcement learning setting. Context-AwareVRD [35] and WeaklySup [23] encode the features of pairs of bounding boxes similarly to Equation (9). However, in Context-AwareVRD the learning is performed with a neural network, whereas in WeaklySup the learning is based on a weakly-supervised discriminative clustering I.e., the supervision on a given relationship is not at triples level but on an image level.…”

Section: Methodsmentioning

confidence: 99%

“…In [2] the background knowledge is statistical information (learnt with statistical link prediction methods [21]) about the training set triples. Contextual information between objects is used also in [23], [35] with different learning methods. In [31] the background knowledge (from the training set and Wikipedia) is a probability distribution of a relationship given the subject/object.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Compensating Supervision Incompleteness with Prior Knowledge in Semantic Image Interpretation

Donadello

Serafini

2019

2019 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

Semantic Image Interpretation is the task of extracting a structured semantic description from images. This requires the detection of visual relationships: triples subject, relation, object describing a semantic relation between a subject and an object. A pure supervised approach to visual relationship detection requires a complete and balanced training set for all the possible combinations of subject, relation, object . However, such training sets are not available and would require a prohibitive human effort. This implies the ability of predicting triples which do not appear in the training set. This problem is called zero-shot learning. State-of-the-art approaches to zero-shot learning exploit similarities among relationships in the training set or external linguistic knowledge. In this paper, we perform zero-shot learning by using Logic Tensor Networks, a novel Statistical Relational Learning framework that exploits both the similarities with other seen relationships and background knowledge, expressed with logical constraints between subjects, relations and objects. The experiments on the Visual Relationship Dataset show that the use of logical constraints outperforms the current methods. This implies that background knowledge can be used to alleviate the incompleteness of training sets.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Compensating Supervision Incompleteness with Prior Knowledge in Semantic Image Interpretation

Donadello

Serafini

2019

2019 International Joint Conference on Neural Networks (IJCNN)

View full text Add to dashboard Cite

show abstract

“…VRD and Visual Genome have spurred the development of new approaches for visual relationship detection. Suc-cessful methods typically build on top of an object detection module, and reason jointly over language and visual features [16,37,19,5,40,35]. Multiple independent directions have been proven fruitful, including learning features that are agnostic to object categories [34], facilitating the interaction between object features and predicate features [34,5,16], overcoming the scarcity of labeled data through weakly supervised learning [27,38], and detecting the relationships among multiple objects jointly as scene graphs [33,18,32,17].…”

Section: Related Workmentioning

confidence: 99%

“…We also report the performance of combining their predictions by a weighted average. We evaluate five state-of-the-art models: Vip-CNN [16], Peyre et al [27], PPR-FCN [40], DRNet [5] and VTransE [37]. They were created for visual relationship detection but can be adapted to our task straightforwardly: First, object detectors are replaced by ground truth objects.…”

Section: Baselines For Spatial Relation Recognitionmentioning

confidence: 99%

SpatialSense: An Adversarially Crowdsourced Benchmark for Spatial Relation Recognition

Yang

Russakovsky

Deng

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Understanding the spatial relations between objects in images is a surprisingly challenging task (Fig. 1). A chair may be "behind" a person even if it appears to the left of the person in the image (depending on which way the person is facing). Two students that appear close to each other in the image may not in fact be "next to" each other if there is a third student between them.We introduce SpatialSense, a dataset specializing in spatial relation recognition which captures a broad spectrum of such challenges, allowing for proper benchmarking of computer vision techniques. SpatialSense is constructed through adversarial crowdsourcing, in which human annotators are tasked with finding spatial relations that are difficult to predict using simple cues such as 2D spatial configuration or language priors. Adversarial crowdsourcing significantly reduces dataset bias and samples more interesting relations in the long tail compared to existing datasets. On SpatialSense, state-of-the-art recognition models perform comparably to simple baselines, suggesting that they rely on straightforward cues instead of fully reasoning about this complex task. The SpatialSense benchmark provides a path forward to advancing the spatial reasoning capabilities of computer vision systems. The dataset and code are available at https://github. com/princeton-vl/SpatialSense.

show abstract

“…Considering the fact that visual features provide limited knowledge for distinguishing the predicates, many works focus on introducing different modal features into predicate recognition stage. For example, [40,41,45,47] prove that language prior and location information denoting categories and location of object pairs are effective to improve the performances of visual relationship recognition. However, comparing to language prior, relative location information, as strong inferring to relationships, are not fully exploited in those works.…”

Section: Related Workmentioning

confidence: 99%

Visual Relationship Detection with Relative Location Mining

Zhou

Zhang

2019

Proceedings of the 27th ACM International Conference on Multimedia

View full text Add to dashboard Cite

Visual relationship detection, as a challenging task used to find and distinguish the interactions between object pairs in one image, has received much attention recently. In this work, we propose a novel visual relationship detection framework by deeply mining and utilizing relative location of object-pair in every stage of the procedure. In both the stages, relative location information of each object-pair is abstracted and encoded as auxiliary feature to improve the distinguishing capability of object-pairs proposing and predicate recognition, respectively; Moreover, one Gated Graph Neural Network(GGNN) is introduced to mine and measure the relevance of predicates using relative location. With the locationbased GGNN, those non-exclusive predicates with similar spatial position can be clustered firstly and then be smoothed with close classification scores, thus the accuracy of top n recall can be increased further. Experiments on two widely used datasets VRD and VG show that, with the deeply mining and exploiting of relative location information, our proposed model significantly outperforms the current state-of-the-art.

show abstract

Towards Context-Aware Interaction Recognition for Visual Relationship Detection

Abstract: The first two authors contributed equally to this work. This work was in part supported by an ARC Future Fellowship to C.

Cited by 155 publications

References 39 publications

Compensating Supervision Incompleteness with Prior Knowledge in Semantic Image Interpretation

Compensating Supervision Incompleteness with Prior Knowledge in Semantic Image Interpretation

SpatialSense: An Adversarially Crowdsourced Benchmark for Spatial Relation Recognition

Visual Relationship Detection with Relative Location Mining

Contact Info

Product

Resources

About