Sentence Directed Video Object Codiscovery

Yu, Haonan; Siskind, Jeffrey Mark

doi:10.1007/s11263-017-1018-6

Cited by 17 publications

(11 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These are limited to small set of nouns. Object co-localization focuses on discovering and detecting an object in images or videos without any bounding box annotation, but only from image/video level labels [3,6,23,30,38,40,51]. These works are similar to ours with respect to the amount of supervision, but they focus on a few discrete classes, while our approach can handle arbitrary phrases and allows for localization of novel phrases.…”

Section: Related Workmentioning

confidence: 99%

Grounding of Textual Phrases in Images by Reconstruction

Rohrbach

et al. 2016

Lecture Notes in Computer Science

435

553

View full text Add to dashboard Cite

Abstract. Grounding (i.e. localizing) arbitrary, free-form textual phrases in visual content is a challenging problem with many applications for human-computer interaction and image-text reference resolution. Few datasets provide the ground truth spatial localization of phrases, thus it is desirable to learn from data with no or little grounding supervision. We propose a novel approach which learns grounding by reconstructing a given phrase using an attention mechanism, which can be either latent or optimized directly. During training our approach encodes the phrase using a recurrent network language model and then learns to attend to the relevant image region in order to reconstruct the input phrase. At test time, the correct attention, i.e., the grounding, is evaluated. If grounding supervision is available it can be directly applied via a loss over the attention mechanism. We demonstrate the effectiveness of our approach on the Flickr 30k Entities [35] and ReferItGame [26] datasets with different levels of supervision, ranging from no supervision over partial supervision to full supervision. Our supervised variant improves by a large margin over the state-of-the-art on both datasets.

show abstract

Section: Related Workmentioning

confidence: 99%

Grounding of Textual Phrases in Images by Reconstruction

Rohrbach

et al. 2016

Lecture Notes in Computer Science

435

553

View full text Add to dashboard Cite

show abstract

“…In recent years, researchers also explore grounding in videos. Yu and Siskind (2015) grounded objects in constrained videos by leveraging weak semantic constraints implied by a sequence of sentences. Vasudevan et al (2018) grounded objects in the last frame of stereo videos with the help of text, motion cues, human gazes and spatial-temporal context.…”

Section: Related Workmentioning

confidence: 99%

Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video

Chen¹,

Ma²,

Luo³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

In this paper, we address a novel task, namely weakly-supervised spatio-temporally grounding natural sentence in video. Specifically, given a natural sentence and a video, we localize a spatio-temporal tube in the video that semantically corresponds to the given sentence, with no reliance on any spatio-temporal annotations during training. First, a set of spatiotemporal tubes, referred to as instances, are extracted from the video. We then encode these instances and the sentence using our proposed attentive interactor which can exploit their fine-grained relationships to characterize their matching behaviors. Besides a ranking loss, a novel diversity loss is introduced to train the proposed attentive interactor to strengthen the matching behaviors of reliable instance-sentence pairs and penalize the unreliable ones. Moreover, we also contribute a dataset, called VID-sentence, based on the Im-ageNet video object detection dataset, to serve as a benchmark for our task. Extensive experimental results demonstrate the superiority of our model over the baseline approaches. Our code and the constructed VID-sentence dataset

show abstract

“…As such the work revisits well-studied terrain (Coyne and Sproat, 2001). Another recent study in this area is Yu and Siskind (2017), wherein spatial relation models are used to locate and identify similar objects in several video streams. We should separately mention the spatial modelling studies by Malinowski and Fritz (2014) and, especially, Collell et al (2017), which apply deep neural networks to learning spatial templates for triplets of form (relatum, relation, referent).…”

Section: Related Workmentioning

confidence: 99%

Computational Models for Spatial Prepositions

Platonov¹,

Schubert²

2018

Proceedings of the First International Workshop on Spatial Language Understanding

View full text Add to dashboard Cite

Developing computational models of spatial prepositions (such as on, in, above, etc.) is crucial for such tasks as human-machine collaboration, story understanding, and 3D model generation from descriptions. However, these prepositions are notoriously vague and ambiguous, with meanings depending on the types, shapes and sizes of entities in the argument positions, the physical and task context, and other factors. As a result truth value judgments for prepositional relations are often uncertain and variable. In this paper we treat the modeling task as calling for assignment of probabilities to such relations as a function of multiple factors, where such probabilities can be viewed as estimates of whether humans would judge the relations to hold in given circumstances. We implemented our models in a 3D blocks world and a room world in a computer graphics setting, and found that true/false judgments based on these models do not differ much more from human judgments that the latter differ from one another. However, what really matters pragmatically is not the accuracy of truth value judgments but whether, for instance, the computer models suffice for identifying objects described in terms of prepositional relations, (e.g., the box to the left of the table, where there are multiple boxes). For such tasks, our models achieved accuracies above 90% for most relations.

show abstract

Sentence Directed Video Object Codiscovery

Cited by 17 publications

References 55 publications

Grounding of Textual Phrases in Images by Reconstruction

Grounding of Textual Phrases in Images by Reconstruction

Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video

Computational Models for Spatial Prepositions

Contact Info

Product

Resources

About