Joint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequences

Abstract

Understanding the structure of complex activities in videos is one of the many challenges faced by action recognition methods. To overcome this challenge, not only do methods need a solid knowledge of the visual structure of underlying features but also a good interpretation of how they could change over time. Consequently, action segmentation tasks must take into account not only the visual cues from individual frames, but their characteristics as a temporal sequence of features.This work presents our findings on the impact of incorporating both visual and temporal learning on an unsupervised action segmentation pipeline. We introduce a novel approach to extract relevant visual and temporal features from untrimmed sequences for the temporal localization of sub-activities within complex actions without any labeling information. Through extensive experimentation on two benchmark datasets -Breakfast Actions, and YouTube Instructions -we show that the proposed approach is able to provide a meaningful visual and temporal embedding from the visual cues from contiguous video frames and that it indeed helps in temporal segmentation.

A Cluster-Based Method for Action Segmentation Using Spatio-Temporal and Positional Encoded Embeddings

“…To address this problem, some previous works included frame-wise feature vectors into a temporal embedding. For instance, VidalMata et al [40] trained a temporal embedding model as a Multilayer Perceptron with the learning goal of predicting the relative timestamp 𝑡 of a given frame.…”

“…Nevertheless, these solutions require frame-level or scene-level annotations that are incredibly laborious. For this reason, researchers started focusing on methods with less supervision, such as weakly-supervised [4,22,29,37] and unsupervised methods [23,33,40].…”

A Cluster-Based Method for Action Segmentation Using Spatio-Temporal and Positional Encoded Embeddings

“…For this reason, researchers started focusing on methods with less supervision, such as weakly-supervised (16,17,18,19) and unsupervised methods (20,21,22). Most of these methods (17,18,19,23,24) are based on the idea of generating pseudo-labels that are used to train supervised models.…”

“…Finally by applying a frame-wise decoding with the Viterbi Algorithm they temporally segment each video by maximizing the probability of a sequence of frames that follows the cluster ordering, this way helping to ensure consistency among the labels for each frame. VidalMatal et al (21) presented an unsupervised approach that segments actions based on visual-temporal embeddings. To achieve this, they do a two stage training method.…”

“…This function generates an embedded space that captures the temporal characteristics for each frame. For instance, VidalMata et al (21) trained a temporal embedding model as a Multilayer Perceptron with the learning goal of predicting the relative timestamp t of a given frame.…”