Gimme Signals: Discriminative signal encoding for multimodal activity recognition

Abstract

We present a simple, yet effective and flexible method for action recognition supporting multiple sensor modalities. Multivariate signal sequences are encoded in an image and are then classified using a recently proposed Effi-cientNet CNN architecture. Our focus was to find an approach that generalizes well across different sensor modalities without specific adaptions while still achieving good results. We apply our method to 4 action recognition datasets containing skeleton sequences, inertial and motion capturing measurements as well as Wi-Fi fingerprints that range up to 120 action classes. Our method defines the current best CNN-based approach on the NTU RGB+D 120 dataset, lifts the state of the art on the ARIL Wi-Fi dataset by +6.78%, improves the UTD-MHAD inertial baseline by +14.43%, the UTD-MHAD skeleton baseline by +1.13% and achieves 96.11% on the Simitate motion capturing data (80/20 split). We further demonstrate experiments on both, modality fusion on a signal level and signal reduction to prevent the representation from overloading.

Action Segmentation on Representations of Skeleton Sequences Using Transformer Networks

“…In order to show generalization of our action segmentation approach we integrate a recently proposed sparse graphbased representation for multi-modal action recognition [8]. In table 5 we compare two such graph representations with similar dense representations.…”

“…The latter is referred to as TSSI in the following sections. Other than these dense representations, we use graphs to convert a skeleton sequence into an image like Memmesheimer et al [8]. Here each coordinate of every joint is graphed in one combining image.…”

“…Our representations are constructed from different building blocks. These include normalization techniques, different coordinate encodings and image assembly methods like those in [1] or [8]. Our goal is to find best-performing combinations of the described techniques and provide an insight into which properties of a representation are beneficial for training.…”

Human Action Recognition from Various Data Modalities: A Review

“…Imran and Raman [284] designed a three-stream network architecture, where a 1D-CNN for gyroscopic data, a 2D-CNN for RGB data, and an RNN for skeleton data were used, and late fusion was adopted to predict the final class label. Memmesheimer et al [389] transformed different modalities as images, and utilized CNN to perform HAR.…”

Fusion-GCN: Multimodal Action Recognition using Graph Convolutional Networks

“…This allows a flexible per modality model-design, but comes at the computational cost of the multiple streams that need to be trained. For early fusion approaches, multiple modalities are fused on a representation level [32], reducing the training process to a single model but potentially loosing the more descriptive features from per-modality models. Kong et al [20] presented a multi modality distillation model.…”