Finding "It": Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos

Huang, De-An; Buch, Shyamal; Dery, Lucio M.; Garg, Animesh; Li, Feifei; Niebles, Juan Carlos

doi:10.1109/cvpr.2018.00623

Cited by 87 publications

(82 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It was first studied in the image domain (Zhao et al, 2018;. Later, given a sequence of transcriptions and their corresponding video clips as well as their temporal alignment, Huang et al (2018) Figure 2: The architecture of our model. An instance generator is used to produce spatio-temporal instances.…”

Section: Related Workmentioning

confidence: 99%

Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video

Chen¹,

Ma²,

Luo³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

In this paper, we address a novel task, namely weakly-supervised spatio-temporally grounding natural sentence in video. Specifically, given a natural sentence and a video, we localize a spatio-temporal tube in the video that semantically corresponds to the given sentence, with no reliance on any spatio-temporal annotations during training. First, a set of spatiotemporal tubes, referred to as instances, are extracted from the video. We then encode these instances and the sentence using our proposed attentive interactor which can exploit their fine-grained relationships to characterize their matching behaviors. Besides a ranking loss, a novel diversity loss is introduced to train the proposed attentive interactor to strengthen the matching behaviors of reliable instance-sentence pairs and penalize the unreliable ones. Moreover, we also contribute a dataset, called VID-sentence, based on the Im-ageNet video object detection dataset, to serve as a benchmark for our task. Extensive experimental results demonstrate the superiority of our model over the baseline approaches. Our code and the constructed VID-sentence dataset

show abstract

Section: Related Workmentioning

confidence: 99%

Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video

Chen¹,

Ma²,

Luo³

et al. 2019

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics

View full text Add to dashboard Cite

show abstract

“…As the ordering supervision can be automatically extracted from language, our work is related to using language as supervision for videos. The supervision usually comes from movie scripts [8,2,41] or transcription of instructional videos [1,33,25,14]. Unlike these approaches, we assume the discrete action labels are already extracted and focus on leveraging the ordering information as supervision.…”

Section: Related Workmentioning

confidence: 99%

D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation

Chang

Huang

Sui

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

Self Cite

158

141

View full text Add to dashboard Cite

We address weakly supervised action alignment and segmentation in videos, where only the order of occurring actions is available during training. We propose Discriminative Differentiable Dynamic Time Warping (D 3 TW), the first discriminative model using weak ordering supervision. The key technical challenge for discriminative modeling with weak supervision is that the loss function of the ordering supervision is usually formulated using dynamic programming and is thus not differentiable. We address this challenge with a continuous relaxation of the min-operator in dynamic programming and extend the alignment loss to be differentiable. The proposed D 3 TW innovatively solves sequence alignment with discriminative modeling and end-toend training, which substantially improves the performance in weakly supervised action alignment and segmentation tasks. We show that our model is able to bypass the degenerated sequence problem usually encountered in previous work and outperform the current state-of-the-art across three evaluation metrics in two challenging datasets.

show abstract

“…In 2017, Huang et al [31] studied the task of reference resolution, which aimed to temporally link an entity to the original action that produced it. In 2018, they further investigated the visual grounding problem [32], which explored the visual-linguistic meaning of referring expressions in both spatial and temporal domains. In the same year, Zhou et al [82] presented a procedure segmentation task, targeting at segmenting an instructional video into category-independent procedure segments.…”

Section: Tasks For Instructional Video Analysismentioning

confidence: 99%

“…As an extension work, they further provided the object level annotation [81]. Both the YouCook and YouCook2 datasets can be used for the video caption tasks, and the YouCook2 further facilitated the research for procedure segmentation [82] and video object grounding [32], [81].…”

Section: Datasetsmentioning

confidence: 99%

Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

Tang

Lu²,

Zhou³

2021

IEEE Trans. Pattern Anal. Mach. Intell.

View full text Add to dashboard Cite

Thanks to the substantial and explosively inscreased instructional videos on the Internet, novices are able to acquire knowledge for completing various tasks. Over the past decade, growing efforts have been devoted to investigating the problem on instructional video analysis. However, the most existing datasets in this area have limitations in diversity and scale, which makes them far from many real-world applications where more diverse activities occur. To address this, we present a large-scale dataset named as "COIN" for COmprehensive INstructional video analysis. Organized with a hierarchical structure, the COIN dataset contains 11,827 videos of 180 tasks in 12 domains (e.g., vehicles, gadgets, etc.) related to our daily life. With a new developed toolbox, all the videos are annotated efficiently with a series of step labels and the corresponding temporal boundaries. In order to provide a benchmark for instructional video analysis, we evaluate plenty of approaches on the COIN dataset under five different settings. Furthermore, we exploit two important characteristics (i.e., task-consistency and ordering-dependency) for localizing important steps in instructional videos. Accordingly, we propose two simple yet effective methods, which can be easily plugged into conventional proposal-based action detection models. We believe the introduction of the COIN dataset will promote the future in-depth research on instructional video analysis for the community. Our dataset, annotation toolbox and source code are available at http://coin-dataset.github.io. DomainTaskStep VehiclesHousehold Items Change the Car Tire {unscrew the screws, jack up the car, remove the tire, put on the tire, tighten the screws } {remove the door knob, remove bolt and pin board, install new pin board, install new bolt, install new door knob } Replace the Door Knob

show abstract

Finding "It": Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos

Cited by 87 publications

References 31 publications

Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video

Weakly-Supervised Spatio-Temporally Grounding Natural Sentence in Video

D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation

Comprehensive Instructional Video Analysis: The COIN Dataset and Performance Evaluation

Contact Info

Product

Resources

About