2-D Skeleton-Based Action Recognition via Two-Branch Stacked LSTM-RNNs

Avola, Danilo; Cascio, Marco; Cinque, Luigi; Foresti, Gian Luca; Massaroni, Cristiano; Rodolà, Emanuele

doi:10.1109/tmm.2019.2960588

Cited by 72 publications

(26 citation statements)

References 78 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Skeleton joint positions can be exploited to produce meaningful features able to describe, for example, gait, using the lower half of the body as shown by several gait re-identification works [ 51 , 52 ], or body emotions, leveraging features extracted from both lower and upper body halves [ 36 ]. Motivated by the encouraging results already obtained via hand-crafted skeleton features in References [ 34 , 35 , 36 ], two meta-feature groups built from skeleton joint positions are proposed in this work to analyze the whole body: local and global meta-features defined

and

, respectively.…”

Section: Methodology: Lstm Hashing Model and Meta-featuresmentioning

confidence: 99%

“…In this paper, a meta-feature based LSTM hashing model for person re-identification is presented. The proposed method takes inspiration from some of our recent experiences in using 2D/3D skeleton based features and LSTM models to recognize hand gestures [ 34 ], body actions [ 35 ], and body affects [ 36 ] in long video sequences. Unlike these, the 2D skeleton model, in this paper, is used to generate biometric features referred to movement, gait, and bone proportions of the body.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Bodyprint—A Meta-Feature Based LSTM Hashing Model for Person Re-Identification

Avola

Cinque

Fagioli

et al. 2020

Sensors

Self Cite

View full text Add to dashboard Cite

Person re-identification is concerned with matching people across disjointed camera views at different places and different time instants. This task results of great interest in computer vision, especially in video surveillance applications where the re-identification and tracking of persons are required on uncontrolled crowded spaces and after long time periods. The latter aspects are responsible for most of the current unsolved problems of person re-identification, in fact, the presence of many people in a location as well as the passing of hours or days give arise to important visual appearance changes of people, for example, clothes, lighting, and occlusions; thus making person re-identification a very hard task. In this paper, for the first time in the state-of-the-art, a meta-feature based Long Short-Term Memory (LSTM) hashing model for person re-identification is presented. Starting from 2D skeletons extracted from RGB video streams, the proposed method computes a set of novel meta-features based on movement, gait, and bone proportions. These features are analysed by a network composed of a single LSTM layer and two dense layers. The first layer is used to create a pattern of the person’s identity, then, the seconds are used to generate a bodyprint hash through binary coding. The effectiveness of the proposed method is tested on three challenging datasets, that is, iLIDS-VID, PRID 2011, and MARS. In particular, the reported results show that the proposed method, which is not based on visual appearance of people, is fully competitive with respect to other methods based on visual features. In addition, thanks to its skeleton model abstraction, the method results to be a concrete contribute to address open problems, such as long-term re-identification and severe illumination changes, which tend to heavily influence the visual appearance of persons.

show abstract

and

, respectively.…”

Section: Methodology: Lstm Hashing Model and Meta-featuresmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Bodyprint—A Meta-Feature Based LSTM Hashing Model for Person Re-Identification

Avola

Cinque

Fagioli

et al. 2020

Sensors

Self Cite

View full text Add to dashboard Cite

show abstract

“…e two-stream or multistream networks used in some researches [8,14,18,20,21,23,30] have achieved good performances, but they have also increased the required numbers of model parameters. erefore, this article embeds bones and the relative positions and motions of joints into a high-dimensional space at the point level and then fuses these three features without multiplying the parameters.…”

Section: Point Levelmentioning

confidence: 99%

“…Since Shahroudy et al [6,7] established the NTU RGB + D data set, a large-scale data set for 3D human activity analyses, deep learning has been widely used in skeleton-based human action recognition studies. e existing research has been divided mainly into two directions: models based on convolutional neural networks(CNNs) [8][9][10][11][12] and models based on recurrent neural networks (RNNs) [13][14][15][16][17][18]. CNN based-methods regard the X, Y, and Z coordinates of joints as image channels, whereas the frame number and joint number of each action sequence are regarded as the length and width of the corresponding image, respectively.…”

Section: Introductionmentioning

confidence: 99%

A Lightweight Hierarchical Model with Frame-Level Joints Adaptive Graph Convolution for Skeleton-Based Action Recognition

Jiang

Yang

Liu

et al. 2021

Security and Communication Networks

View full text Add to dashboard Cite

In skeleton-based human action recognition methods, human behaviours can be analysed through temporal and spatial changes in the human skeleton. Skeletons are not limited by clothing changes, lighting conditions, or complex backgrounds. This recognition method is robust and has aroused great interest; however, many existing studies used deep-layer networks with large numbers of required parameters to improve the model performance and thus lost the advantage of less computation of skeleton data. It is difficult to deploy previously established models to real-life applications based on low-cost embedded devices. To obtain a model with fewer parameters and a higher accuracy, this study designed a lightweight frame-level joints adaptive graph convolutional network (FLAGCN) model to solve skeleton-based action recognition tasks. Compared with the classical 2s-AGCN model, the new model obtained a higher precision with 1/8 of the parameters and 1/9 of the floating-point operations (FLOPs). Our proposed network characterises three main improvements. First, a previous feature-fusion method replaces the multistream network and reduces the number of required parameters. Second, at the spatial level, two kinds of graph convolution methods capture different aspects of human action information. A frame-level graph convolution constructs a human topological structure for each data frame, whereas an adjacency graph convolution captures the characteristics of the adjacent joints. Third, the model proposed in this study hierarchically extracts different levels of action sequence features, making the model clear and easy to understand; further, it reduces the depth of the model and the number of parameters. A large number of experiments on the NTU RGB + D 60 and 120 data sets show that this method has the advantages of few required parameters, low computational costs, and fast speeds. It also has a simple structure and training process that make it easy to deploy in real-time recognition systems based on low-cost embedded devices.

show abstract

“…In spatial domain the models extract features with respect to the pixel location in image space using models such as Convolutional Neural Networks (CNNs) [2], [7]. For temporal or time series modelling of the RGB D data, Recurrent neural networks (RNNs) and their upgrades such as Long Short-Term Memory (LSTM) nets [15], [16].…”

Section: Introductionmentioning

confidence: 99%