Self-supervised Video Representation Learning with Cross-Stream Prototypical Contrasting

Abstract

Instance-level contrastive learning techniques, which rely on data augmentation and a contrastive loss function, have found great success in the domain of visual representation learning. They are not suitable for exploiting the rich dynamical structure of video however, as operations are done on many augmented instances. In this paper we propose "Video Cross-Stream Prototypical Contrasting", a novel method which predicts consistent prototype assignments from both RGB and optical flow views, operating on sets of samples. Specifically, we alternate the optimization process; while optimizing one of the streams, all views are mapped to one set of stream prototype vectors. Each of the assignments is predicted with all views except the one matching the prediction, pushing representations closer to their assigned prototypes. As a result, more efficient video embeddings with ingrained motion information are learned, without the explicit need for optical flow computation during inference. We obtain state-of-the-art results on nearestneighbour video retrieval and action recognition, outperforming previous best by +3.2% on UCF101 using the S3D backbone (90.5% Top-1 acc), and by +7.2% on UCF101 and +15.1% on HMDB51 using the R(2+1)D backbone. 1 * Work is done during an internship at BrainCreators, Amsterdam.

Comparing Learning Methodologies for Self-Supervised Audio-Visual Representation Learning

“…In the same line of the SimCLR framework, Han et al [19] have proposed a new self-supervised co-training method called CoCLR, in which contrastive learning with positive samples mining is used to align RGB and optical flow data. Toering et al combine the co-training mechanism and the idea of the SwAV approach [12] to propose prototypical contrastive learning [18] as an alternative to instance-based contrastive learning. VOLUME 10, 2022 Similarly, audio data can provide a useful information about the content of videos.…”