Attention-Driven Body Pose Encoding for Human Activity Recognition

Abstract

This article proposes a novel attention-based body pose encoding for human activity recognition that presents a enriched representation of body-pose that is learned. The enriched data complements the 3D body joint position data and improves model performance. In this paper, we propose a novel approach that learns enhanced feature representations from a given sequence of 3D body joints. To achieve this encoding, the approach exploits 1) a spatial stream which encodes the spatial relationship between various body joints at each time point to learn spatial structure involving the spatial distribution of different body joints 2) a temporal stream that learns the temporal variation of individual body joints over the entire sequence duration to present a temporally enhanced representation. Afterwards, these two pose streams are fused with a multi-head attention mechanism. We also capture the contextual information from the RGB video stream using a Inception-ResNet-V2 model combined with a multi-head attention and a bidirectional Long Short-Term Memory (LSTM) network. Finally, the RGB video stream is combined with the fused body pose stream to give a novel end-to-end deep model for effective human activity recognition.

A Step Towards Automated Functional Assessment of Activities of Daily Living

“…Thus, authors have focused on enriching the pose-information with physics-based measurements such velocities and acceleration [7,42], different normalisation techniques [42], relative body joints positions [15] etc. Instead of handcrafting such features, [6] uses SEU and TEU to automatically learn enhanced representations that can capture structural information and various inter-joint dependencies of the human body joints. Inspired by this, we adapt the TCN-ResNet model [16] to use a spatial-temporal architecture involving SEU and TEU layer to advance the light-weight human activity recognition approaches.…”

“…1.2: The proposed model consists of a spatial and a temporal stream where each stream uses a TCN-ResNet [16]. Block-A of the spatial stream is replaced with the SEU [6] while the same block in temporal stream is exchanged with the TEU [6]. The GAP layer of the TCN-ResNet [16] is replaced by a FV-based pooling mechanism [24].…”

“…1.2). As in [6], for each frame the SEU captures the structural relationship between various body joints and enhanced/augmented representation of the human body pose sequence to rest of the network. On the other hand, the goal of the TEU is to encode the frame-wise positions of body joints and present a temporally rich representation for each joint, individually.…”

“…The difference is, it adapts the Block-A (Fig. 1.2) to the SEU introduced by [6]. Normal 1D convolutions process all the frames in a body-pose sequence together as shown in Eq.…”

“…Furthermore, we present a novel pose-based two-stream multi-label activity recognition model based on TCN-ResNet [16] that comprehensively outperforms the TCN-ResNet on our dataset and the well-known NTU-RGBD ADL recognition dataset [28]. The two-stream architecture inspired by [6], which consists of a spatial and a temporal stream. The spatial stream contains a Spatial Encoding Unit (SEU), which provides an enriched representation that learns to capture the structural relationships between various body joints in a given video frame.…”