LSTM Self-Supervision for Detailed Behavior Analysis

Brattoli, Biagio; Büchler, Uta; Wahl, Anna-Sophia; Ommer, Björn

doi:10.1109/cvpr.2017.399

Cited by 53 publications

(42 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the context of video data, the temporal structure of video data has been exploited to fine-tune networks on train-1 https://github.com/annusha/unsup_temp_embed ing data without labels [34,2]. The temporal ordering of video frames has also been used to learn feature representations for action recognition [20,23,9,4].…”

Section: Related Workmentioning

confidence: 99%

Unsupervised Learning of Action Classes With Continuous Temporal Embedding

Kukleva

Kuehne

Sener

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

112

160

View full text Add to dashboard Cite

The task of temporally detecting and segmenting actions in untrimmed videos has seen an increased attention recently. One problem in this context arises from the need to define and label action boundaries to create annotations for training which is very time and cost intensive. To address this issue, we propose an unsupervised approach for learning action classes from untrimmed video sequences. To this end, we use a continuous temporal embedding of framewise features to benefit from the sequential nature of activities. Based on the latent space created by the embedding, we identify clusters of temporal segments across all videos that correspond to semantic meaningful action classes. The approach is evaluated on three challenging datasets, namely the Breakfast dataset, YouTube Instructions, and the 50Salads dataset. While previous works assumed that the videos contain the same high level activity, we furthermore show that the proposed approach can also be applied to a more general setting where the content of the videos is unknown.

show abstract

Section: Related Workmentioning

confidence: 99%

Unsupervised Learning of Action Classes With Continuous Temporal Embedding

Kukleva

Kuehne

Sener

et al. 2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

112

160

View full text Add to dashboard Cite

show abstract

“…Other types of pretext tasks proposed for unsupervised learning include understanding the correct order of video frames [6,36] or predicting the spatial relation between image patches [12], e.g., jigsaw puzzle solving as a pretext task was exploited by Noorozi and Favaro [37]. In another work, Noroozi et al [38] proposed to train an unsupervised model by counting the primitive elements of images.…”

Section: Related Workmentioning

confidence: 99%

Self-Supervised Representation Learning via Neighborhood-Relational Encoding

Sabokrou

Khalooei

Adeli

2019

2019 IEEE/CVF International Conference on Computer Vision (ICCV)

View full text Add to dashboard Cite

In this paper, we propose a novel self-supervised representation learning by taking advantage of a neighborhoodrelational encoding (NRE) among the training data. Conventional unsupervised learning methods only focused on training deep networks to understand the primitive characteristics of the visual data, mainly to be able to reconstruct the data from a latent space. They often neglected the relation among the samples, which can serve as an important metric for self-supervision. Different from the previous work, NRE aims at preserving the local neighborhood structure on the data manifold. Therefore, it is less sensitive to outliers. We integrate our NRE component with an encoder-decoder structure for learning to represent samples considering their local neighborhood information. Such discriminative and unsupervised representation learning scheme is adaptable to different computer vision tasks due to its independence from intense annotation requirements. We evaluate our proposed method for different tasks, including classification, detection, and segmentation based on the learned latent representations. In addition, we adopt the auto-encoding capability of our proposed method for applications like defense against adversarial example attacks and video anomaly detection. Results confirm the performance of our method is better or at least comparable with the state-of-the-art for each specific application, but with a generic and self-supervised approach.

show abstract

“…Instead of the binary task of tuple verification like Misra et al [32], our self-supervised task is to predict the exact permutation of the patches, much like the jigsaw puzzle task of Noroozi and Favaro [34] only on videos. Some recent approaches have used temporal coherency-based self-supervision on video sequences to model fine-grained human poses and activities [31] and animal behavior [7]. Our model is not specialized for motor skill learning like [7] and we do not require bounding boxes for humans in the video frames as in [31].…”

Section: Related Workmentioning

confidence: 99%

“…Some recent approaches have used temporal coherency-based self-supervision on video sequences to model fine-grained human poses and activities [31] and animal behavior [7]. Our model is not specialized for motor skill learning like [7] and we do not require bounding boxes for humans in the video frames as in [31].…”

Section: Related Workmentioning

confidence: 99%

Video Jigsaw: Unsupervised Learning of Spatiotemporal Context for Video Action Recognition

Ahsan

Madhok

Essa

2019

2019 IEEE Winter Conference on Applications of Computer Vision (WACV)

View full text Add to dashboard Cite

We propose a self-supervised learning method to jointly reason about spatial and temporal context for video recognition. Recent self-supervised approaches have used spatial context [9,34] as well as temporal coherency [32] but a combination of the two requires extensive preprocessing such as tracking objects through millions of video frames [59] or computing optical flow to determine frame regions with high motion [30]. We propose to combine spatial and temporal context in one self-supervised framework without any heavy preprocessing. We divide multiple video frames into grids of patches and train a network to solve jigsaw puzzles on these patches from multiple frames. So the network is trained to correctly identify the position of a patch within a video frame as well as the position of a patch over time. We also propose a novel permutation strategy that outperforms random permutations while significantly reducing computational and memory constraints. We use our trained network for transfer learning tasks such as video activity recognition and demonstrate the strength of our approach on two benchmark video action recognition datasets without using a single frame from these datasets for unsupervised pretraining of our proposed video jigsaw network.

show abstract

LSTM Self-Supervision for Detailed Behavior Analysis

Cited by 53 publications

References 22 publications

Unsupervised Learning of Action Classes With Continuous Temporal Embedding

Unsupervised Learning of Action Classes With Continuous Temporal Embedding

Self-Supervised Representation Learning via Neighborhood-Relational Encoding

Video Jigsaw: Unsupervised Learning of Spatiotemporal Context for Video Action Recognition

Contact Info

Product

Resources

About