Breaking Shortcut: Exploring Fully Convolutional Cycle-Consistency for Video Correspondence Learning

Tang, Yun; Jiang, Zhenyu; Xie, Zhenda; Cao, Yue; Zhang, Zheng; Torr, Philip H. S.; Hu, Han

doi:10.48550/arxiv.2105.05838

Cited by 3 publications

(3 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Jabri et al [25] posed cycle-consistency as a random walk problem, allowing the model to obtain dense supervision from multiframe video. Tang et al [65] proposed an extension that allowed for fully convolutional training. These approaches are trained on sparse patches and learn coarse-grained correspondences.…”

Section: Related Workmentioning

confidence: 99%

Learning Pixel Trajectories with Multiscale Contrastive Random Walks

Bian¹,

Jabri²,

Efros³

et al. 2022

Preprint

View full text Add to dashboard Cite

A range of video modeling tasks, from optical flow to multiple object tracking, share the same fundamental challenge: establishing space-time correspondence. Yet, approaches that dominate each space differ. We take a step towards bridging this gap by extending the recent contrastive random walk formulation to much denser, pixel-level spacetime graphs. The main contribution is introducing hierarchy into the search problem by computing the transition matrix between two frames in a coarse-to-fine manner, forming a multiscale contrastive random walk when extended in time. This establishes a unified technique for selfsupervised learning of optical flow, keypoint tracking, and video object segmentation. Experiments demonstrate that, for each of these tasks, the unified model achieves performance competitive with strong self-supervised approaches specific to that task.

show abstract

Section: Related Workmentioning

confidence: 99%

Learning Pixel Trajectories with Multiscale Contrastive Random Walks

Bian¹,

Jabri²,

Efros³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The key idea is that after tracking a target from a given startpoint to an estimated endpopint, reversing the video and re-applying the tracker from the endpoint should lead back to the original startpoint. This core idea is typically combined with patchlevel affinity matrices, which can be traversed with spatial transformers [11], region-level motion averages [10], or random walks [12,23,24]. Unlike existing self-supervision works which initialize weights from ImageNet [25] or randomly, our method begins with state-of-the-art models whose architectures and weights are pre-optimized for motion estimation [13,14] and attempts to further refine them on a new test domain.…”

Section: Related Workmentioning

confidence: 99%

Relation Extraction on the Wargame based on Pre-trained Models

Sun¹,

Qi²

2022

2022 9th International Conference on Dependable Systems and Their Applications (DSA)

View full text Add to dashboard Cite

Given the difficulty of manually annotating motion in video, the current best motion estimation methods are trained with synthetic data, and therefore struggle somewhat due to a train/test gap. Self-supervised methods hold the promise of training directly on real video, but typically perform worse. These include methods trained with warp error (i.e., color constancy) combined with smoothness terms, and methods that encourage cycle-consistency in the estimates (i.e., tracking backwards should yield the opposite trajectory as tracking forwards). In this work, we take on the challenge of improving state-of-the-art supervised models with self-supervised training. We find that when the initialization is supervised weights, most existing self-supervision techniques actually make performance worse instead of better, which suggests that the benefit of seeing the new data is overshadowed by the noise in the training signal. Focusing on obtaining a "clean" training signal from real-world unlabelled video, we propose to separate labelmaking and training into two distinct stages. In the first stage, we use the pre-trained model to estimate motion in a video, and then select the subset of motion estimates which we can verify with cycle-consistency. This produces a sparse but accurate pseudo-labelling of the video. In the second stage, we fine-tune the model to reproduce these outputs, while also applying augmentations on the input. We complement this boot-strapping method with simple techniques that densify and re-balance the pseudo-labels, ensuring that we do not merely train on "easy" tracks. We show that our method yields reliable gains over fully-supervised methods in real videos, for both short-term (flow-based) and long-range (multiframe) pixel tracking. Our code can be found here: https: //github.com/AlexSunNik/refining-motion-code.

show abstract

“…The Space-Time Memory Network [35] memorizes intermediate frames with segmentation masks as references and performs pixellevel matching between them with the current frame to segment target objects in a bottom-up manner, which has been proved effective and has served as the current mainstream framework. Some works [40,23,5,15,59,41,51,6,62,46,25,27] further develop STM and have achieved excellent performance.…”

Section: Introductionmentioning

confidence: 99%

Global Spectral Filter Memory Network for Video Object Segmentation

Liu¹,

Rong²,

Wang³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper studies semi-supervised video object segmentation through boosting intra-frame interaction. Recent memory networkbased methods focus on exploiting inter-frame temporal reference while paying little attention to intra-frame spatial dependency. Specifically, these segmentation model tends to be susceptible to interference from unrelated nontarget objects in a certain frame. To this end, we propose Global Spectral Filter Memory network (GSFM), which improves intraframe interaction through learning long-term spatial dependencies in the spectral domain. The key components of GSFM is 2D (inverse) discrete Fourier transform for spatial information mixing. Besides, we empirically find low frequency feature should be enhanced in encoder (backbone) while high frequency for decoder (segmentation head). We attribute this to semantic information extracting role for encoder and fine-grained details highlighting role for decoder. Thus, Low (High) Frequency Module is proposed to fit this circumstance. Extensive experiments on the popular DAVIS and YouTube-VOS benchmarks demonstrate that GSFM noticeably outperforms the baseline method and achieves state-of-theart performance. Besides, extensive analysis shows that the proposed modules are reasonable and of great generalization ability. Our source code is available at https://github.com/workforai/GSFM.

show abstract

Breaking Shortcut: Exploring Fully Convolutional Cycle-Consistency for Video Correspondence Learning

Cited by 3 publications

References 34 publications

Learning Pixel Trajectories with Multiscale Contrastive Random Walks

Learning Pixel Trajectories with Multiscale Contrastive Random Walks

Relation Extraction on the Wargame based on Pre-trained Models

Global Spectral Filter Memory Network for Video Object Segmentation

Contact Info

Product

Resources

About