Gimme Signals: Discriminative signal encoding for multimodal activity recognition

Memmesheimer, Raphael; Theisen, Nick; Paulus, Dietrich

doi:10.48550/arxiv.2003.06156

Cited by 5 publications

(15 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Imran and Raman [284] designed a three-stream network architecture, where a 1D-CNN for gyroscopic data, a 2D-CNN for RGB data, and an RNN for skeleton data were used, and late fusion was adopted to predict the final class label. Memmesheimer et al [389] transformed different modalities as images, and utilized CNN to perform HAR.…”

Section: Fusion Of Visual and Sensor Modalitiesmentioning

confidence: 99%

Human Action Recognition from Various Data Modalities: A Review

Sun,

Ke,

Rahmani

et al. 2020

Preprint

View full text Add to dashboard Cite

Human Action Recognition (HAR), aiming to understand human behaviors and then assign category labels, has a wide range of applications, and thus has been attracting increasing attention in the field of computer vision. Generally, human actions can be represented using various data modalities, such as RGB, skeleton, depth, infrared sequence, point cloud, event stream, audio, acceleration, radar, and WiFi, etc., which encode different sources of useful yet distinct information and have various advantages and application scenarios. Consequently, lots of existing works have attempted to investigate different types of approaches for HAR using various modalities. In this paper, we give a comprehensive survey for HAR from the perspective of the input data modalities. Specifically, we review both the hand-crafted feature-based and deep learning-based methods for single data modalities, and also review the methods based on multiple modalities, including the fusion-based frameworks and the co-learning-based approaches. The current benchmark datasets for HAR are also introduced. Finally, we discuss some potentially important research directions in this area.

show abstract

Section: Fusion Of Visual and Sensor Modalitiesmentioning

confidence: 99%

Human Action Recognition from Various Data Modalities: A Review

Sun,

Ke,

Rahmani

et al. 2020

Preprint

View full text Add to dashboard Cite

show abstract

“…Occlusion could not be directly tackled, therefore the representation by Liu, Liu, and Chen [14] added flexibility by fusing up to nine representation schemes in separate image channels. A similar representation has recently shown to be usable also for action recognition on different modalities and their fusion [15]. Kim and Reiter [7] on the other hand presented a compact and humaninterpretable representation.…”

Section: Related Workmentioning

confidence: 99%

“…4 gives a concrete example from a throw sequence. In contrast to the representations used for multimodal action recognition [15] or skeleton based action recognition [25,14] the proposed representation is more compact.…”

Section: B Representationsmentioning

confidence: 99%

“…In total 24 models were trained for this ablation study. We trained these models for 200 epochs Skepxel [29] 29.6 SkeleMotion Orientation [1] 34.4 TSSI [1] 41.0 Gimme Signals [15] 41.5 SkeleMotion Magnitude [1] 44.4 SL-DML [16] 50.9 Ours 54.2…”

Section: Ablation Studymentioning

confidence: 99%

“…To support the effectiveness of our proposed representation in a metric learning setting we compare against other skeleton-based image representations. We use the publicly avail able implementation for the SkeleMotion [1], SL-DML [16], Gimme Signals [15] and re-implementations of the TSSI [29] and Skepxels [9] representations to integrate them into our metric learning approach. These representations have been described in Section II more detailed.…”

Section: Comparison With Related Representationsmentioning

confidence: 99%

See 2 more Smart Citations

Skeleton-DML: Deep Metric Learning for Skeleton-Based One-Shot Action Recognition

Memmesheimer¹,

Haring²,

Theisen³

et al. 2020

Preprint

Self Cite

View full text Add to dashboard Cite

One-shot action recognition allows the recognition of human-performed actions with only a single training example. This can influence human-robot-interaction positively by enabling the robot to react to previously unseen behaviour. We formulate the one-shot action recognition problem as a deep metric learning problem and propose a novel imagebased skeleton representation that performs well in a metric learning setting. Therefore, we train a model that projects the image representations into an embedding space. In embedding space the similar actions have a low euclidean distance while dissimilar actions have a higher distance. The one-shot action recognition problem becomes a nearest-neighbor search in a set of activity reference samples. We evaluate the performance of our proposed representation against a variety of other skeletonbased image representations. In addition, we present an ablation study that shows the influence of different embedding vector sizes, losses and augmentation. Our approach lifts the state-ofthe-art by +3.3% for the one-shot action recognition protocol on the NTU RGB+D 120 dataset under a comparable training setup. With additional augmentation our result improved over +7.7%.

show abstract