Visual Relationship Detection with Internal and External Linguistic Knowledge Distillation

Abstract

Understanding the visual relationship between two objects involves identifying the subject, the object, and a predicate relating them. We leverage the strong correlations between the predicate and the subj, obj pair (both semantically and spatially) to predict predicates conditioned on the subjects and the objects. Modeling the three entities jointly more accurately reflects their relationships compared to modeling them independently, but it complicates learning since the semantic space of visual relationships is huge and training data is limited, especially for longtail relationships that have few instances. To overcome this, we use knowledge of linguistic statistics to regularize visual model learning. We obtain linguistic knowledge by mining from both training annotations (internal knowledge) and publicly available text, e.g., Wikipedia (external knowledge), computing the conditional probability distribution of a predicate given a subj, obj pair. As we train the visual model, we distill this knowledge into the deep model to achieve better generalization. Our experimental results on the Visual Relationship Detection (VRD) and Visual Genome datasets suggest that with this linguistic knowledge distillation, our model outperforms the stateof-the-art methods significantly, especially when predicting unseen relationships (e.g., recall improved from 8.45% to 19.17% on VRD zero-shot testing set).

Compensating Supervision Incompleteness with Prior Knowledge in Semantic Image Interpretation

“…Contextual information between objects is used also in [23], [35] with different learning methods. In [31] the background knowledge (from the training set and Wikipedia) is a probability distribution of a relationship given the subject/object. This knowledge drives the learning of visual relationships.…”

“…Visual relationships are mainly detected with supervised learning techniques [31]. These require large training sets of images annotated with bounding boxes and relationships [14], [19].…”

Visual Relationship Detection with Relative Location Mining

“…Considering the fact that visual features provide limited knowledge for distinguishing the predicates, many works focus on introducing different modal features into predicate recognition stage. For example, [40,41,45,47] prove that language prior and location information denoting categories and location of object pairs are effective to improve the performances of visual relationship recognition. However, comparing to language prior, relative location information, as strong inferring to relationships, are not fully exploited in those works.…”

“…R@n computes the Recall using the top n object-pairs proposals' predictions in one image. Following [40], we also set a hyperparameter k, which means to take the top k predictions into consideration per object-pair. In visual relationship detection task, R@n,k=1 is equivalent to R@n in [22].…”

“…However, this kind of object-pairs proposing scheme faces one big challenge: how to select M reasonable object-pairs from N 2 possible combinations? Some works [16,40] attempt to reserve specific object-pairs proposals based on the objectiveness scores from detection model. But there is a large gap between the higher objectiveness scores and the more meaningful object-pairs, which deteriorates performances inevitably in this proposing scheme.…”