Skeleton-DML: Deep Metric Learning for Skeleton-Based One-Shot Action Recognition

Memmesheimer, Raphael; Haring, S.; Theisen, Nick; Paulus, Dietrich

doi:10.1109/wacv51458.2022.00091

Cited by 33 publications

(30 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our PROFORMER model yields state-of-the-art on three datasets [15], [16], [26], surpassing the best published approach on the challenging NTU-120 benchmark by 1.84% for one-shot action recognition. 4) As a side-observation, we discover that PROFORMER optimization strategy is much more resistant to noise corruptions, outperforming the same backbone trained with conventional deep metric learning strategy.…”

Section: Introductionmentioning

confidence: 91%

“…This is impractical in real robotics applications, where data-efficient learning of new concepts on-the-fly remains a key challenge [9]. Problems of learning activity representations which adapt well to new data-scarce categories are often posed in the form of few-shot recognition, where the methods usually fall into one of two categories: (1) meta-learning-based methods [18], [31], [32], [33], which reinitialize a new set of tasks every epoch following the "learning to learn" paradigm and (2) metric-learning-based methods [15], [16], [17], which aim to project the input to a lower-dimensional space, where same-category samples are close to each other and the intercategory ones are far apart. Our approach falls in the latter category.…”

Section: Related Workmentioning

confidence: 99%

“…Few-shot learning has been well-studied in object detection [34], [35] and various recognition tasks [36], [37]. The research of one-shot activity recognition from 3D body poses has been much more sparse and is largely studied in the context of one-shot recognition on the NTU-120 dataset [15], [26], [38], [39], [40]. State-of-the-art recognition results are currently reached by the approach of Memmesheimer et al [16], which uses a CNN-based encoding of 3D skeletons represented as images and optimizes the framework with deep metric learning using a mixture of cross entropy and triplet margin losses.…”

Section: Related Workmentioning

confidence: 99%

“…State-of-the-art recognition results are currently reached by the approach of Memmesheimer et al [16], which uses a CNN-based encoding of 3D skeletons represented as images and optimizes the framework with deep metric learning using a mixture of cross entropy and triplet margin losses. While we specifically focus on learning data-efficient representations of 3D body poses [15], we need to acknowledge an array of work on video-based activity recognition from few training examples [11], [13]. In this work, we for the first time explore the potential of encoding body pose sequences directly using visual transformers, which has been addressed with CNN-based [15] encoders in the past.…”

Section: Related Workmentioning

confidence: 99%

“…Problems of data-efficient ADL recognition are often posed in the form of one-or few-shot learning [11], [12], [13], [14] and addressed with metric learning [15], [16], [17] or meta learning [18]. For example, the state-of-the-art approaches for one-shot action recognition from body pose conduct a systematic study featuring off-the-shelf transformer architectures benchmarked in the tasks of skeletonbased one-shot activity recognition.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

ProFormer: Learning Data-efficient Representations of Body Movement with Prototype-based Feature Augmentation and Visual Transformers

Peng¹,

Roitberg²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Automatically understanding human behaviour allows household robots to identify the most critical needs and plan how to assist the human according to the current situation. However, the majority of such methods are developed under the assumption that a large amount of labelled training examples is available for all concepts-of-interest. Robots, on the other hand, operate in constantly changing unstructured environments, and need to adapt to novel action categories from very few samples. Methods for data-efficient recognition from body poses increasingly leverage skeleton sequences structured as image-like arrays and then used as input to convolutional neural networks. We look at this paradigm from the perspective of transformer networks, for the first time exploring visual transformers as data-efficient encoders of skeleton movement. In our pipeline, body pose sequences cast as image-like representations are converted into patch embeddings and then passed to a visual transformer backbone optimized with deep metric learning. Inspired by recent success of feature enhancement methods in semi-supervised learning, we further introduce PROFORMER -an improved training strategy which uses soft-attention applied on iteratively estimated action category PROtotypes used to augment the embeddings and compute an auxiliary consistency loss. Extensive experiments consistently demonstrate the effectiveness of our approach for one-shot recognition from body poses, achieving state-of-the-art results on multiple datasets and surpassing the best published approach on the challenging NTU-120 one-shot benchmark by 1.84%. Our code will be made publicly available at https: //github.com/KPeng9510/ProFormer.

show abstract

Section: Introductionmentioning

confidence: 91%