Aligning Videos in Space and Time

Purushwalkam, Senthil; Tian, Ye; Gupta, Saurabh; Gupta, Abhinav

doi:10.1007/978-3-030-58574-7_16

Cited by 21 publications

(14 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The learned representation has shown to be useful for multiple downstream recogni-tion and geometry estimation tasks. Instead of learning a general representation, there is a line of recent research on self-supervised learning specifically for finding correspondence [70,74,71,41,40,43,34,59]. For example, Vondrick et al [70] propose to propagate the current frame color to predict the future frame color as a pretext task to learn fine-grained correspondence between the current and future frame.…”

Section: Related Workmentioning

confidence: 99%

“…For example, Vondrick et al [70] propose to propagate the current frame color to predict the future frame color as a pretext task to learn fine-grained correspondence between the current and future frame. Other tasks including tracking objects [74,71] and patches [34,59] are also designed to explicitly find different levels of correspondences. But is explicit tracking task the only way to learn correspondence?…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective

Wang

2021

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

Learning a good representation for space-time correspondence is the key for various computer vision tasks, including tracking object bounding boxes and performing video object pixel segmentation. To learn generalizable representation for correspondence in large-scale, a variety of self-supervised pretext tasks are proposed to explicitly perform object-level or patch-level similarity learning. Instead of following the previous literature, we propose to learn correspondence using Video Frame-level Similarity (VFS) learning, i.e, simply learning from comparing video frames. Our work is inspired by the recent success in image-level contrastive learning and similarity learning for visual recognition. Our hypothesis is that if the representation is good for recognition, it requires the convolutional features to find correspondence between similar objects or parts. Our experiments show surprising results that VFS surpasses state-of-the-art self-supervised approaches for both OTB visual object tracking and DAVIS video object segmentation. We perform detailed analysis on what matters in VFS and reveals new properties on image and frame level similarity learning. Project page is available at https://jerryxu.net/VFS.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective

Wang

2021

2021 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

show abstract

“…Cycle-consistency learning. Our work is influenced by cycle-consistency learning in different computer vision applications including 3D scene understanding [31,73,20,71], image alignment and translation [74,76,75,77], and space-time alignment in videos [4,62,38,17,59,32,49,37]. For example, Wang et al [62] propose to perform forward and backward tracking in time to achieve a cycleconsistency for learning temporal correspondence.…”

Section: Related Workmentioning

confidence: 99%

“…Dwibedi et al [17] formulates a temporal cycle consistency loss which aligns frames from one video to another between a pair of videos, and achieves good performance in video frame alignemnt tasks. Building on these two works, Purushwalkam et al [49] propose to track object patches inside a video and align them across videos at the same time. While these results are encouraging, both approaches learning from video pairs [17,49] require human annota-tors to provide ground-truth pairs (video-level) in training with a small scale of videos.…”

Section: Related Workmentioning

confidence: 99%

“…Building on these two works, Purushwalkam et al [49] propose to track object patches inside a video and align them across videos at the same time. While these results are encouraging, both approaches learning from video pairs [17,49] require human annota-tors to provide ground-truth pairs (video-level) in training with a small scale of videos. In this paper, we propose to go beyond these restrictions, and apply cross-video cycleconsistency learning without any human annotations.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency

Wu¹,

Wang²

2021

Preprint

View full text Add to dashboard Cite

Recent works have advanced the performance of selfsupervised representation learning by a large margin. The core among these methods is intra-image invariance learning. Two different transformations of one image instance are considered as a positive sample pair, where various tasks are designed to learn invariant representations by comparing the pair. Analogically, for video data, representations of frames from the same video are trained to be closer than frames from other videos, i.e. intravideo invariance. However, cross-video relation has barely been explored for visual representation learning. Unlike intra-video invariance, ground-truth labels of crossvideo relation is usually unavailable without human labors. In this paper, we propose a novel contrastive learning method which explores the cross-video relation by using cycle-consistency for general image representation learning. This allows to collect positive sample pairs across different video instances, which we hypothesize will lead to higher-level semantics. We validate our method by transferring our image representation to multiple downstream tasks including visual object tracking, image classification, and action recognition. We show significant improvement over state-of-the-art contrastive learning methods. Project page is available at https://happywu.github.io/ cycle_contrast_video .

show abstract

Bridging Images and Videos: A Simple Learning Framework for Large Vocabulary Video Object Detection

Woo

Park

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Aligning Videos in Space and Time

Cited by 21 publications

References 53 publications

Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective

Rethinking Self-supervised Correspondence Learning: A Video Frame-level Similarity Perspective

Contrastive Learning of Image Representations with Cross-Video Cycle-Consistency

Bridging Images and Videos: A Simple Learning Framework for Large Vocabulary Video Object Detection

Contact Info

Product

Resources

About