Representation Learning via Global Temporal Alignment and Cycle-Consistency

Hadji, Isma; Derpanis, Konstantinos G.; Jepson, Allan D.

doi:10.1109/cvpr46437.2021.01092

Cited by 47 publications

(29 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…• Our framework outperforms the state-of-the-art methods by a large margin on multiple tasks across different datasets. For example, under the linear evaluation protocol on FineGym [37] dataset, our framework achieves 41.75% accuracy, which is +13.94% higher than the existing best method GTA [21]. On Penn-Action [50] dataset, our method achieves 91.67% for fine-grained classification, 99.1% for Kendall's Tau, and 90.58% top-5 accuracy for fine-grained frame retrieval, which all surpass the existing best methods.…”

Section: Liquid Starts Exitingmentioning

confidence: 82%

“…representations to predict the action category. In contrast, many practical applications, e.g., sign language translation [4,5,13], robotic imitation learning [29,36], action alignment [6,21,23] and phase classification [16,27,37,50] require algorithms having ability to model long videos with hundreds of frames and extract frame-wise representations rather than the global features (Fig. 1).…”

Section: Liquid Starts Exitingmentioning

confidence: 99%

“…Previous methods [27,35,37] have made an effort to learn frame-wise representations via supervised learning, where sub-actions or phase boundaries are annotated. However, it is time-consuming and even impractical to manually label each frame and exact action boundaries [21] on largescale datasets, which hinders the generalization of models trained with fully supervised learning in realistic scenarios. To reduce the dependency of labeled data, some methods such as TCC [16], LAV [23] and GTA [21] explored weakly-supervised learning by using either cycleconsistency loss [16] or soft dynamic time warping [21,23].…”

Section: Liquid Starts Exitingmentioning

confidence: 99%

“…However, it is time-consuming and even impractical to manually label each frame and exact action boundaries [21] on largescale datasets, which hinders the generalization of models trained with fully supervised learning in realistic scenarios. To reduce the dependency of labeled data, some methods such as TCC [16], LAV [23] and GTA [21] explored weakly-supervised learning by using either cycleconsistency loss [16] or soft dynamic time warping [21,23]. All these methods rely on video-level annotations and the training is conducted on the paired videos describing the same action.…”

Section: Liquid Starts Exitingmentioning

confidence: 99%

See 3 more Smart Citations

Frame-wise Action Representations for Long Videos via Sequence Contrastive Learning

Chen¹,

Wei²,

Li³

2022

Preprint

View full text Add to dashboard Cite

Prior works on action representation learning mainly focus on designing various architectures to extract the global representations for short video clips. In contrast, many practical applications such as video alignment have strong demand for learning dense representations for long videos. In this paper, we introduce a novel contrastive action representation learning (CARL) framework to learn frame-wise action representations, especially for long videos, in a selfsupervised manner. Concretely, we introduce a simple yet efficient video encoder that considers spatio-temporal context to extract frame-wise representations. Inspired by the recent progress of self-supervised learning, we present a novel sequence contrastive loss (SCL) applied on two correlated views obtained through a series of spatio-temporal data augmentations. SCL optimizes the embedding space by minimizing the KL-divergence between the sequence similarity of two augmented views and a prior Gaussian distribution of timestamp distance. Experiments on FineGym, PennAction and Pouring datasets show that our method outperforms previous state-of-the-art by a large margin for downstream fine-grained action classification. Surprisingly, although without training on paired videos, our approach also shows outstanding performance on video alignment and fine-grained frame retrieval tasks. Code and models are available at https://github.com/ minghchen/CARL_code.

show abstract

Section: Liquid Starts Exitingmentioning

confidence: 82%

Section: Liquid Starts Exitingmentioning

confidence: 99%

Section: Liquid Starts Exitingmentioning

confidence: 99%

Section: Liquid Starts Exitingmentioning

confidence: 99%

See 2 more Smart Citations

Frame-wise Action Representations for Long Videos via Sequence Contrastive Learning

Chen¹,

Wei²,

Li³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…[7] uses global sequence alignment as a proxy task by relying on the DTW. [12,20] extended the DTW for end-to-end learning with differentiable approximations of the discrete operations (e.g., the 'min' operator) in the DTW. Chang et al [6] proposed the frame-wise alignment loss using the DTW in weakly supervised action alignment in videos.…”

Section: Related Workmentioning

confidence: 99%

Video-Text Representation Learning via Differentiable Weak Temporal Alignment

Ko¹,

Choi²,

Ko³

et al. 2022

Preprint

View full text Add to dashboard Cite

Learning generic joint representations for video and text by a supervised method requires a prohibitively substantial amount of manually annotated video datasets. As a practical alternative, a large-scale but uncurated and narrated video dataset, HowTo100M, has recently been introduced. But it is still challenging to learn joint embeddings of video and text in a self-supervised manner, due to its ambiguity and non-sequential alignment. In this paper, we propose a novel multi-modal self-supervised framework Video-Text Temporally Weak Alignment-based Contrastive Learning (VT-TWINS) to capture significant information from noisy and weakly correlated data using a variant of Dynamic Time Warping (DTW). We observe that the standard DTW inherently cannot handle weakly correlated data and only considers the globally optimal alignment path. To address these problems, we develop a differentiable DTW which also reflects local information with weak temporal alignment. Moreover, our proposed model applies a contrastive learning scheme to learn feature representations on weakly correlated data. Our extensive experiments demonstrate that VT-TWINS attains significant improvements in multi-modal representation learning and outperforms various challenging downstream tasks. Code is available at https://github.com/mlvlab/VT-TWINS.

show abstract