Broaden Your Views for Self-Supervised Video Learning

Recasens, Adrià; Luc, Pauline; Alayrac, Jean-Baptiste; Wang, Luyu; Strub, Florian; Tallec, Corentin; Malinowski, Mateusz; Patraaucean, Viorica; Altché, Florent; Valko, Michal; Grill, Jean-Bastien; Oord, Aäron van den; Zisserman, Andrew

doi:10.1109/iccv48922.2021.00129

Cited by 84 publications

(45 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some examples include temporal consistency [190], [191], interframe predictability [192], geometric transformations [193], motion statistics [194] or playback speed [195], [196]. Moreover, there has been a recent surge of self-supervised vision tasks, such as SimCLR [182] and Barlow Twins [197] for images or BraVE [198] for video. Some such method, DINO [199], has achieved impressive results for image Transformers.…”

Section: The Road Aheadmentioning

confidence: 99%

Video Transformers: A Survey

Selva¹,

Johansen²,

Escalera³

et al. 2022

Preprint

View full text Add to dashboard Cite

Transformer models have shown great success modeling long-range interactions. Nevertheless, they scale quadratically with input length and lack inductive biases. These limitations can be further exacerbated when dealing with the high dimensionality of video. Proper modeling of video, which can span from seconds to hours, requires handling long-range interactions. This makes Transformers a promising tool for solving video related tasks, but some adaptations are required. While there are previous works that study the advances of Transformers for vision tasks, there is none that focus on in-depth analysis of video-specific designs. In this survey we analyse and summarize the main contributions and trends for adapting Transformers to model video data. Specifically, we delve into how videos are embedded and tokenized, finding a very widspread use of large CNN backbones to reduce dimensionality and a predominance of patches and frames as tokens. Furthermore, we study how the Transformer layer has been tweaked to handle longer sequences, generally by reducing the number of tokens in single attention operation. Also, we analyse the self-supervised losses used to train Video Transformers, which to date are mostly constrained to contrastive approaches. Finally, we explore how other modalities are integrated with video and conduct a performance comparison on the most common benchmark for Video Transformers (i.e., action classification), finding them to outperform 3D CNN counterparts with equivalent FLOPs and no significant parameter increase.

show abstract

Section: The Road Aheadmentioning

confidence: 99%

Video Transformers: A Survey

Selva¹,

Johansen²,

Escalera³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Self-supervised Learning in Videos. While selfsupervised learning in videos were initially dominated by approaches based on pretext tasks unique to the video domain [2,31,42,53,55,61,69,[78][79][80]83] recent work focuses more on contrastive losses similar to the image domain [21,30,34,35,63,64]. A combination of previous Our video attention block takes a 3D tensor and applies self-attention along temporal and spatial dimension.…”

Section: Related Workmentioning

confidence: 99%

“…Unlike these works, we jointly vary spatial and temporal resolutions and use a predictive objective as self-supervision. The idea of views with limited locality is explored in [8,64]. While [8] uses views of varying locality for disentangling the representation space into temporally local and global features using contrastive objectives, our approach uses view locality to learn correspondences along and across dimensions with our predictive objective.…”

Section: Related Workmentioning

confidence: 99%

“…Recent self-supervised methods perform on-par with supervised learning for certain vision tasks [11,12,16,36]. The necessity of self-supervised learning is even greater in domains such as video analysis where annotations are more expensive [39,43,62,64].…”

Section: Introductionmentioning

confidence: 99%

“…Many existing self-supervised representation learning methods on videos [63,64,87] use contrastive learning objectives which can require larger batch sizes, longer training regimes, careful negative mining and dedicated memory banks. Further, the contrastive objectives require careful temporal sampling [63] and multiple networks looking at similar/different clips to develop attract/repel loss formulations [64]. In contrast, we propose to learn self-supervised features from unlabelled videos via self-distillation [71] by a twin network strategy (student-teacher models) [13,33].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Self-supervised Video Transformer

Ranasinghe¹,

Naseer²,

Khan³

et al. 2021

Preprint

View full text Add to dashboard Cite

In this paper, we propose self-supervised training for video transformers using unlabelled video data. From a given video, we create local and global spatiotemporal views with varying spatial sizes and frame rates. Our selfsupervised objective seeks to match the features of these different views representing the same video, to be invariant to spatiotemporal variations in actions. To the best of our knowledge, the proposed approach is the first to alleviate the dependency on negative samples or dedicated memory banks in Self-supervised Video Transformer (SVT). Further, owing to the flexibility of Transformer models, SVT supports slow-fast video processing within a single architecture using dynamically adjusted positional encodings and supports long-term relationship modeling along spatiotemporal dimensions. Our approach performs well on four action recognition benchmarks (Kinetics-400, UCF-101, HMDB-51, and SSv2) and converges faster with small batch sizes.

show abstract