Self-supervised Video Representation Learning with Cross-Stream Prototypical Contrasting

Toering, Martine; Gatopoulos, Ioannis; Stol, Maarten; Hu, Vincent Tao

doi:10.1109/wacv51458.2022.00092

Cited by 15 publications

(28 citation statements)

References 42 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…2) context-based models that consider the contextual features of data, such as spatial or temporal relationships between different parts of each sample or inter-sample similarity (e.g. clustering methods); 3) semantic-based models that are trained based on automatically generated semantic labels, such as depth of image and moving objects and 4) cross-modality-based models in which the pretext tasks are to verify whether the two channels of input data are relevant/synchronised or not, such as RGB-flow [139], visual-audio [8] or visual-text [112] correspondence correlation. However, this classification scheme is specifically designed for studies on CV problems.…”

Section: Background and Frameworkmentioning

confidence: 99%

Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Data

Deldari¹,

Xue²,

Saeed³

et al. 2022

Preprint

View full text Add to dashboard Cite

Recently, Self-Supervised Representation Learning (SSRL) has attracted much attention in the field of computer vision, speech, natural language processing (NLP), and recently, with other types of modalities, including time series from sensors. The popularity of self-supervised learning is driven by the fact that traditional models typically require a huge amount of well-annotated data for training. Acquiring annotated data can be a difficult and costly process. Self-supervised methods have been introduced to improve the efficiency of training data through discriminative pre-training of models using supervisory signals that have been freely obtained from the raw data. Unlike existing reviews of SSRL that have pre-dominately focused upon methods in the fields of CV or NLP for a single modality, we aim to provide the first comprehensive review of multimodal self-supervised learning methods for temporal data.To this end, we 1) provide a comprehensive categorization of existing SSRL methods, 2) introduce a generic pipeline by defining the key components of a SSRL framework, 3) compare existing models in terms of their objective function, network architecture and potential applications, and 4) review existing multimodal techniques in each category and various modalities. Finally, we present existing weaknesses and future opportunities. We believe our work develops a perspective on the requirements of SSRL in domains that utilise multimodal and/or temporal data.

show abstract

Section: Background and Frameworkmentioning

confidence: 99%

Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Data

Deldari¹,

Xue²,

Saeed³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…This intuitive work motivated researchers to use optical flow as a representation of motion for activity understanding [12,30,71,72], and they have achieved significant improvements over RGB-only models in the supervised learning literature. Inspired by this success, many recent SSL works [33,37,80] have explored using optical flow (OF) to advance SSL beyond RGB-only baselines. Han et al (CoCLR) [37] used OF to retrieve positive samples for the infoNCA [63], which led to significant improvements.…”

Section: Introductionmentioning

confidence: 99%

“…Nevertheless, CoCLR did not utilize OF for training the backbone and hence may not have realized the full potential of learning motion representations. VICC [80] adopted the online cluster assignment [11] to videos by considering OF as another view of RGB, and minimized the distance between RGB and OF features during online clustering. This method obtained SOTA results on various datasets.…”

Section: Introductionmentioning

confidence: 99%

“…However, enforcing similarity between RGB and OF features can be detrimental, especially if one information source is noisy, as is usually the case for OF due to camera motion. Furthermore, these mod-els [37,80] require a complicated training strategy that successively updates one model while freezing the parameters of the other model, which prevents end-to-end training.…”

Section: Introductionmentioning

confidence: 99%

“…After we obtain our final assignment, we train a backbone network by minimizing a cross entropy loss between the final cluster assignment of different augmentations of the same video (as used in [11]). The proposed idea has several benefits compared to the state of the art [11,37,80]. First, it constructs more robust clusters during training due to prior information, which is particularly important when one information source is noisy.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

GOCA: Guided Online Cluster Assignment for Self-Supervised Video Representation Learning

Coskun¹,

Zareian²,

Moore³

et al. 2022

Preprint

View full text Add to dashboard Cite

Clustering is a ubiquitous tool in unsupervised learning. Most of the existing self-supervised representation learning methods typically cluster samples based on visually dominant features. While this works well for image-based self-supervision, it often fails for videos, which require understanding motion rather than focusing on background. Using optical flow as complementary information to RGB can alleviate this problem. However, we observe that a naïve combination of the two views does not provide meaningful gains. In this paper, we propose a principled way to combine two views. Specifically, we propose a novel clustering strategy where we use the initial cluster assignment of each view as prior to guide the final cluster assignment of the other view. This idea will enforce similar cluster structures for both views, and the formed clusters will be semantically abstract and robust to noisy inputs coming from each individual view. Additionally, we propose a novel regularization strategy to address the feature collapse problem, which is common in cluster-based self-supervised learning methods. Our extensive evaluation shows the effectiveness of our learned representations on downstream tasks, e.g., video retrieval and action recognition. Specifically, we outperform the state of the art by 7% on UCF and 4% on HMDB for video retrieval, and 5% on UCF and 6% on HMDB for video classification. 1

show abstract

GOCA: Guided Online Cluster Assignment for Self-supervised Video Representation Learning

Coskun

Zareian

Moore

et al. 2022

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Self-supervised Video Representation Learning with Cross-Stream Prototypical Contrasting

Cited by 15 publications

References 42 publications

Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Data

Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Data

GOCA: Guided Online Cluster Assignment for Self-Supervised Video Representation Learning

GOCA: Guided Online Cluster Assignment for Self-supervised Video Representation Learning

Contact Info

Product

Resources

About