Combining CNN streams of RGB-D and skeletal data for human activity recognition

Khaire, Pushpajit; Kumar, Praveen; Imran, Javed

doi:10.1016/j.patrec.2018.04.035

Cited by 131 publications

(70 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In addition, from Table 5 we observe that the proposed method achieves comparable performance compared with method [68] in which learned features from three modalities are fused in feature-level. The method [70] leads to the stateof-the-art performance since it fuses information collected from four types of sensors, i.e., RGB camera, depth sensor, and two wearable inertial sensors (accelerometer and gyroscope data) for action recognition.…”

Section: E Utd-mhad Dataset and Performance Evaluationmentioning

confidence: 88%

Multi-Modal Human Action Recognition With Sub-Action Exploiting and Class-Privacy Preserved Collaborative Representation Learning

Liang

Liu

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Multimodal human action recognition with depth sensors has drawn wide attention, due to its potential applications such as health-care monitoring, smart buildings/home, intelligent transportation, and security surveillance. As one of the obstacles of robust action recognition, sub-actions sharing, especially among similar action categories, makes human action recognition more challenging. This paper proposes a segmental architecture to exploit the relations of sub-actions, jointly with heterogeneous information fusion and Class-privacy Preserved Collaborative Representation (CPPCR) for multi-modal human action recognition. Specifically, a segmental architecture is proposed based on the normalized action motion energy. It models long-range temporal structure over video sequences to better distinguish the similar actions bearing sub-action sharing phenomenon. The sub-action based depth motion and skeleton features are then extracted and fused. Moreover, by introducing within-class local consistency into Collaborative Representation (CR) coding, CPPCR is proposed to address the challenging sub-action sharing phenomenon, learning the high-level discriminative representation. Experiments on four datasets demonstrate the effectiveness of the proposed method.

show abstract

Section: E Utd-mhad Dataset and Performance Evaluationmentioning

confidence: 88%

Multi-Modal Human Action Recognition With Sub-Action Exploiting and Class-Privacy Preserved Collaborative Representation Learning

Liang

Liu

et al. 2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…where Figure 4: The steps of the appropriate frame region selection and extraction of the skeleton motion [16], [37] I X and I Y , are the derivative of the image with respect to X and Y , respectively. In order to obtain these derivatives, horizontal and vertical Sobel filters (i.e.…”

Section: B) Bgs and Human Action Identificationmentioning

confidence: 99%

“…During the process of feature extraction to display action, a combination of contour-based distance signal feature, flow-based motion feature [12], [14], and uniform rotation local binary patterns can be used to define region of interest for feature extraction [15], [16], [17], [22]. Therefore, at this stage, suitable regions for extraction of the feature are determined.…”

Section: D) Roi Calculationmentioning

confidence: 99%

Complex Human Action Recognition in Live Videos Using Hybrid FR-DL Method

Serpush¹,

Rezaei²

2020

Preprint

View full text Add to dashboard Cite

Automated human action recognition is one of the most attractive and practical research fields in computer vision, in spite of its high computational costs. In such systems, the human action labelling is based on the appearance and patterns of the motions in the video sequences; however, the conventional methodologies and classic neural networks cannot use temporal information for action recognition prediction in the upcoming frames in a video sequence. On the other hand, the computational cost of the preprocessing stage is high. In this paper, we address challenges of the preprocessing phase, by an automated selection of representative frames among the input sequences. Furthermore, we extract the key features of the representative frame rather than the entire features. We propose a hybrid technique using background subtraction and HOG, followed by application of a deep neural network and skeletal modelling method. The combination of a CNN and the LSTM recursive network is considered for feature selection and maintaining the previous information, and finally, a Softmax-KNN classifier is used for labelling human activities. We name our model as "Feature Reduction \& Deep Learning" based action recognition method, or FR-DL in short. To evaluate the proposed method, we use the UCF dataset for the benchmarking which is widely-used among researchers in action recognition research. The dataset includes 101 complicated activities in the wild. Experimental results show a significant improvement in terms of accuracy and speed in comparison with six state-of-the-art articles.

show abstract

“…Khaire et al [14] proposed a ConvNets based approach for activity recognition by combining multiple deep learning methods's cues . A new method of creating skeleton images, from skeleton joint sequences, representing motion information is also presented in this paper.…”

Section: Related Workmentioning

confidence: 99%

Contextual data classification for a ubiquitous intelligent environment

et al. 2020

View full text Add to dashboard Cite

Recognition of activities from sensors is a key paradigm of ubiquitous computing. Activity recognition systems can be used to label large sets of data. Variability in human activities, sensor deployment characteristics, and application domains has led to the development of best practices and methods to improve the robustness of activity recognition systems. Classification is one of the most important steps in making the recognition process more expressive and reducing uncertainty, thus minimizing representation. The K-medoid algorithm is simple but effective for grouping data according to the similarity that the samples present between them without the need to know each sample's membership class. In this paper, we propose a classification technique based on unsupervised partitioning algorithm, which allows recognizing activities and overcomes the problem of supervision.

show abstract

Combining CNN streams of RGB-D and skeletal data for human activity recognition

Cited by 131 publications

References 21 publications

Multi-Modal Human Action Recognition With Sub-Action Exploiting and Class-Privacy Preserved Collaborative Representation Learning

Multi-Modal Human Action Recognition With Sub-Action Exploiting and Class-Privacy Preserved Collaborative Representation Learning

Complex Human Action Recognition in Live Videos Using Hybrid FR-DL Method

Contextual data classification for a ubiquitous intelligent environment

Contact Info

Product

Resources

About