A Compact Sequence Encoding Scheme for Online Human Activity Recognition in HRI Applications

Abstract

Human activity recognition and analysis has always been one of the most active areas of pattern recognition and machine intelligence, with applications in various fields, including but not limited to exertion games, surveillance, sports analytics and healthcare. Especially in Human-Robot Interaction, human activity understanding plays a crucial role as household robotic assistants are a trend of the near future. However, state-of-the-art infrastructures that can support complex machine intelligence tasks are not always available, and may not be for the average consumer, as robotic hardware is expensive. In this paper we propose a novel action sequence encoding scheme which efficiently transforms spatio-temporal action sequences into compact representations, using Mahalanobis distance-based shape features and the Radon transform. This representation can be used as input for a lightweight convolutional neural network. Experiments show that the proposed pipeline, when based on state-of-the-art human pose estimation techniques, can provide a robust end-to-end online action recognition scheme, deployable on hardware lacking extreme computing capabilities.

Learning Spatio-Temporal Radon Footprints for Assessment of Parkinson’s Dyskinesia

“…The proposed methodology is based on the application of a robust feature extraction scheme paired with a Convolutional Neural Network (CNN). Specifically, this approach entails the utilization of Spatio-temporal Radon Footprints (SRF), which was initially presented in [13]. We introduce a novel action sequence encoding scheme, which efficiently transforms spatio-temporal information into compact representations, using Mahalanobis distance-based shape features and the Radon transform.…”

“…Given that each person performs an action in a unique way, compared to any other, we ensure that the network is never trained and evaluated on the same person's data. LOPOCV is considered a rigorous cross-validation protocol, suitable for an action recognition and assessment task [13], particularly one in which the dataset is not considerably large. It may be a strict protocol, but it fully responds to the reality and nature of the problem.…”

“…These variations depend on the nature of movements and their chronological occurrences. More detail on the aforementioned sequence encoding scheme can be found in [13]. More specifically, a sinogram is a visual representation derived from the Radon transform, offering a unique perspective on an image's internal structures and features.…”

“…Figure 1. Calculating SRFt: the Mahalanobis matrix is formulated from the corresponding Mahalanobis distances for each one of the depicted joints as the action sequence progresses and the video frames pass until a specific time point t; the Radon transform is applied on the matrix producing a sinogram which consists a compact representation of action's information up to the corresponding moment in time [13]. This procedure is extended across all the timesteps of an action sequence, so different SRFs are produced as an action progresses.…”

“…To achieve this, we introduce a novel methodology centered on encoding the movement sequence by generating 2D sinusoidal representations, the so-called Spatio-temporal Radon Footprints (SRF) [13], which contain all the necessary information, i.e., all the extracted features. Then, we use them for appropriately training Convolutional Neural Networks (CNN) in order to predict the rating values according to the UDysRS metric.…”