SL-DML: Signal Level Deep Metric Learning for Multimodal One-Shot Action Recognition

ProFormer: Learning Data-efficient Representations of Body Movement with Prototype-based Feature Augmentation and Visual Transformers

“…Our PROFORMER model yields state-of-the-art on three datasets [15], [16], [26], surpassing the best published approach on the challenging NTU-120 benchmark by 1.84% for one-shot action recognition. 4) As a side-observation, we discover that PROFORMER optimization strategy is much more resistant to noise corruptions, outperforming the same backbone trained with conventional deep metric learning strategy.…”

“…This is impractical in real robotics applications, where data-efficient learning of new concepts on-the-fly remains a key challenge [9]. Problems of learning activity representations which adapt well to new data-scarce categories are often posed in the form of few-shot recognition, where the methods usually fall into one of two categories: (1) meta-learning-based methods [18], [31], [32], [33], which reinitialize a new set of tasks every epoch following the "learning to learn" paradigm and (2) metric-learning-based methods [15], [16], [17], which aim to project the input to a lower-dimensional space, where same-category samples are close to each other and the intercategory ones are far apart. Our approach falls in the latter category.…”

“…The research of one-shot activity recognition from 3D body poses has been much more sparse and is largely studied in the context of one-shot recognition on the NTU-120 dataset [15], [26], [38], [39], [40]. State-of-the-art recognition results are currently reached by the approach of Memmesheimer et al [16], which uses a CNN-based encoding of 3D skeletons represented as images and optimizes the framework with deep metric learning using a mixture of cross entropy and triplet margin losses. While we specifically focus on learning data-efficient representations of 3D body poses [15], we need to acknowledge an array of work on video-based activity recognition from few training examples [11], [13].…”

“…Problems of data-efficient ADL recognition are often posed in the form of one-or few-shot learning [11], [12], [13], [14] and addressed with metric learning [15], [16], [17] or meta learning [18]. For example, the state-of-the-art approaches for one-shot action recognition from body pose conduct a systematic study featuring off-the-shelf transformer architectures benchmarked in the tasks of skeletonbased one-shot activity recognition.…”

3D Skeleton-based Few-shot Action Recognition with JEANIE is not so Naïve

“…With an exception of very recent models [53,54,59,60], FSAR approaches that learn from skeleton-based 3D body joints are scarce. The above situation prevails despite AR from articulated sets of connected body joints, expressed as 3D coordinates, does offer a number of advantages over videos such as (i) the lack of the background clutter, (ii) the volume of data being several orders of magnitude smaller, and (iii) the 3D geometric manipulations of sequences being relatively friendly.…”