Unsupervised Representation Learning by Sorting Sequences

Lee, Hsin-Ying; Huang, Jia-Bin; Singh, Maneesh; Yang, Ming–Hsuan

doi:10.1109/iccv.2017.79

Cited by 570 publications

(452 citation statements)

References 32 publications

(70 reference statements)

Supporting

Mentioning

452

Contrasting

Order By: Relevance

“…The results are given in Table 4, four phenomena can be observed: First, when self-supervised training with only UCF101, our DPC (60.6%) outperforms all previous methods under similar settings. Note that OPN [22] performs worse when input resolution increases, which indicates a simple self-supervised task like order prediction may not capture the rich semantics from videos. Second, when using Kinetics-400 for self-supervised pre-training, our DPC (68.2%) outperforms all the previous methods by a large margin.…”

Section: Comparison With State-of-the-art Methodsmentioning

confidence: 99%

Video Representation Learning by Dense Predictive Coding

Han

Xie

Zisserman

2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

379

302

View full text Add to dashboard Cite

Figure 1: Nearest Neighbour (NN) video clip retrieval on UCF101. Each row contains four video clips, a query clip and the top three retrievals using clip embeddings. To get the embedding, each video is passed to a 3D-ResNet18, average pooled to a single vector, and cosine similarity is used for retrieval. (a) Embeddings obtained by Dense Predictive Coding (DPC); (b) Embeddings obtained by using the inflated ImageNet pretrained weights. The DPC captures the semantics of the human action, rather than the scene appearance or layout as captured by the ImageNet trained embeddings. In the DPC retrievals the actual appearances of frames can vary dramatically, e.g. in the change in camera viewpoint for the climbing case. AbstractThe objective of this paper is self-supervised learning of spatio-temporal embeddings from video, suitable for human action recognition.We make three contributions: First, we introduce the Dense Predictive Coding (DPC) framework for selfsupervised representation learning on videos. This learns a dense encoding of spatio-temporal blocks by recurrently predicting future representations; Second, we propose a curriculum training scheme to predict further into the future with progressively less temporal context. This encourages the model to only encode slowly varying spatialtemporal signals, therefore leading to semantic representations; Third, we evaluate the approach by first training the DPC model on the Kinetics-400 dataset with selfsupervised learning, and then finetuning the representation on a downstream task, i.e. action recognition. With single stream (RGB only), DPC pretrained representations achieve state-of-the-art self-supervised performance on both UCF101 (75.7% top1 acc) and HMDB51 (35.7% top1 acc), outperforming all previous learning methods by a significant margin, and approaching the performance of a baseline pre-trained on ImageNet. The code is available at https

show abstract

Section: Comparison With State-of-the-art Methodsmentioning

confidence: 99%

Video Representation Learning by Dense Predictive Coding

Han

Xie

Zisserman

2019

2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW)

379

302

View full text Add to dashboard Cite

show abstract

“…In the context of video data, the temporal structure of video data has been exploited to fine-tune networks on train-1 https://github.com/annusha/unsup_temp_embed ing data without labels [34,2]. The temporal ordering of video frames has also been used to learn feature representations for action recognition [20,23,9,4]. Lee et al [20] learn a video representation in an unsupervised manner by solving a sequence sorting problem.…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Learning of Action Classes With Continuous Temporal Embedding

Kukleva

Kuehne

Sener

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

112

160

View full text Add to dashboard Cite

The task of temporally detecting and segmenting actions in untrimmed videos has seen an increased attention recently. One problem in this context arises from the need to define and label action boundaries to create annotations for training which is very time and cost intensive. To address this issue, we propose an unsupervised approach for learning action classes from untrimmed video sequences. To this end, we use a continuous temporal embedding of framewise features to benefit from the sequential nature of activities. Based on the latent space created by the embedding, we identify clusters of temporal segments across all videos that correspond to semantic meaningful action classes. The approach is evaluated on three challenging datasets, namely the Breakfast dataset, YouTube Instructions, and the 50Salads dataset. While previous works assumed that the videos contain the same high level activity, we furthermore show that the proposed approach can also be applied to a more general setting where the content of the videos is unknown.

show abstract

“…In addition, these methods are often lin-ear, and/or treat the data as cross-sectional, and thus do not exploit non-linear relationships. Self-supervised learning was already successfully applied to timeseries video data in the field of computer vision, where Lee et al developed a solution based on time-shuffling [5], which inspired our method.…”

Section: Introductionmentioning

confidence: 99%

Modeling Disease Progression in Retinal OCTs with Longitudinal Self-supervised Learning

Rivail

Schmidt‐Erfurth

Vogl

et al. 2019

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Longitudinal imaging is capable of capturing the static anatomical structures and the dynamic changes of the morphology resulting from aging or disease progression. Self-supervised learning allows to learn new representation from available large unlabelled data without any expert knowledge. We propose a deep learning self-supervised approach to model disease progression from longitudinal retinal optical coherence tomography (OCT). Our self-supervised model takes benefit from a generic time-related task, by learning to estimate the time interval between pairs of scans acquired from the same patient. This task is (i) easy to implement, (ii) allows to use irregularly sampled data, (iii) is tolerant to poor registration, and (iv) does not rely on additional annotations. This novel method learns a representation that focuses on progression specific information only, which can be transferred to other types of longitudinal problems. We transfer the learnt representation to a clinically highly relevant task of predicting the onset of an advanced stage of age-related macular degeneration within a given time interval based on a single OCT scan. The boost in prediction accuracy, in comparison to a network learned from scratch or transferred from traditional tasks, demonstrates that our pretrained self-supervised representation learns a clinically meaningful information.

show abstract

Unsupervised Representation Learning by Sorting Sequences

Cited by 570 publications

References 32 publications

Video Representation Learning by Dense Predictive Coding

Video Representation Learning by Dense Predictive Coding

Unsupervised Learning of Action Classes With Continuous Temporal Embedding

Modeling Disease Progression in Retinal OCTs with Longitudinal Self-supervised Learning

Contact Info

Product

Resources

About