Human Motion Analysis with Deep Metric Learning

Abstract

Effectively measuring the similarity between two human motions is necessary for several computer vision tasks such as gait analysis, person identification and action retrieval. Nevertheless, we believe that traditional approaches such as L2 distance or Dynamic Time Warping based on hand-crafted local pose metrics fail to appropriately capture the semantic relationship across motions and, as such, are not suitable for being employed as metrics within these tasks. This work addresses this limitation by means of a triplet-based deep metric learning specifically tailored to deal with human motion data, in particular with the problem of varying input size and computationally expensive hard negative mining due to motion pair alignment. Specifically, we propose (1) a novel metric learning objective based on a triplet architecture and Maximum Mean Discrepancy; as well as, (2) a novel deep architecture based on attentive recurrent neural networks. One benefit of our objective function is that it enforces a better separation within the learned embedding space of the different motion categories by means of the associated distribution moments. At the same time, our attentive recurrent neural network allows processing varying input sizes to a fixed size of embedding while learning to focus on those motion parts that are semantically distinctive. Our experiments on two different datasets demonstrate significant improvements over conventional human motion metrics.

Transformers in Action: Weakly Supervised Action Segmentation

“…Sentence Embedding/Retrieval: Existing works have considered the task of defining metrics for quantifying, ranking and retrieving similar images [34], images with specific properties [35] or hierarchical similarities [20], [36]. A deep metric learning approach [37], [38] is often taken, using contrastive or discriminative methods [19] to train a similarity metric into the embedding. For application into videos, action segmentation methods have often considered embedding at the frame level to retrieve frames with similar actions in unsupervised [39], [40] methods, or recently the aforementioned DTW-based transcript prototypes [8].…”

Learning Human Activity From Visual Data Using Deep Learning

“…It's worth noting that visual data is only one type of information that can be utilized to recognize activities. RGB-D data in the context of deep learning to distinguish human actions is an example of how various types can be used [19], [20]. Sudhakaran et al [21] also published a hierarchical feature lightweight aggregation approach that can be integrated into any deep architecture with a CNN backbone in another project.…”

Action Recognition Using Local Visual Descriptors and Inertial Data

“…On the other hand only Visual data was used in action recognition. For instance, RGB-D data have been used in deep learning to recognize human actions [13,22]. Sudhakaran et al [35] presented a hierarchical feature lightweight aggregation scheme that can be plugged into any deep architecture with CNN backbone.…”