Improving Visual Relationship Detection Using Semantic Modeling of Scene Descriptions

Abstract

Structured scene descriptions of images are useful for the automatic processing and querying of large image databases. We show how the combination of a semantic and a visual statistical model can improve on the task of mapping images to their associated scene description. In this paper we consider scene descriptions which are represented as a set of triples (subject, predicate, object), where each triple consists of a pair of visual objects, which appear in the image, and the relationship between them (e.g. man-riding-elephant, man-wearing-hat). We combine a standard visual model for object detection, based on convolutional neural networks, with a latent variable model for link prediction. We apply multiple state-of-the-art link prediction methods and compare their capability for visual relationship detection. One of the main advantages of link prediction methods is that they can also generalize to triples, which have never been observed in the training data. Our experimental results on the recently published Stanford Visual Relationship dataset, a challenging real world dataset, show that the integration of a semantic model using link prediction methods can significantly improve the results for visual relationship detection. Our combined approach achieves superior performance compared to the state-of-the-art method from the Stanford computer vision group.

Compensating Supervision Incompleteness with Prior Knowledge in Semantic Image Interpretation

“…These scores are combined with a language prior score (based on word embeddings) that models the semantics of the visual relationships. The methods in [2] also combines visual and semantic information. However, link prediction methods (RESCAL, MultiwayNN, CompleEx, DistMult) are used for modelling the visual relationship semantics in place of word embeddings.…”

“…The visual knowledge consists in the features of the union of the subject and object bounding boxes. In [2] the background knowledge is statistical information (learnt with statistical link prediction methods [21]) about the training set triples. Contextual information between objects is used also in [23], [35] with different learning methods.…”

A Boxology of Design Patterns forHybrid Learningand Reasoning Systems

“…An example of this are the Logic Tensor Networks in [46], where the authors show that encoding prior knowledge in symbolic form allows for better learning results on fewer training data, as well as more robustness against noise. A similar example is given in [47], where knowledge graphs are successfully used as priors in a scene description task, and in [48] where logical rules are used as background knowledge for a gradient descent learning task in a high-dimensional real-valued vector space.…”

Deeply Supervised Multimodal Attentional Translation Embeddings for Visual Relationship Detection

“…Visual Appearance Features are extracted from the predicate box, i.e. the minimum rectangle that encompasses the subject box and the object box [1,12,2,13,14,3], the separate subject-object boxes [5,15,16,17], or both [18,19,4,20,21]. All the above train a single branch with visual features, while we jointly train two separate branches with different features, a predicate feature branch (P-branch) and an object-subject branch (OS-branch), and employ Deep Supervision to align their scores into a common space.…”

“…Linguistic and Semantic Features are employed in a feature-level integration with word embeddings [1,13,4,12,3], encoding of statistics [18,13,4,14], late-fusion with subject-object classemes (score vectors) [22,5,14,19] and loss-level fusion as regularization terms [1,3] or adaptive-margins [4,19,12]. Closest to us, [2] uses subject-object embeddings to train context-aware classifiers and [20] trains multimodal embeddings by projecting visual and linguistic features into a common space.…”