A union of deep learning and swarm-based optimization for 3D human action recognition

Abstract

Human Action Recognition (HAR) is a popular area of research in computer vision due to its wide range of applications such as surveillance, health care, and gaming, etc. Action recognition based on 3D skeleton data allows simplistic, cost-efficient models to be formed making it a widely used method. In this work, we propose DSwarm-Net, a framework that employs deep learning and swarm intelligence-based metaheuristic for HAR that uses 3D skeleton data for action classification. We extract four different types of features from the skeletal data namely: Distance, Distance Velocity, Angle, and Angle Velocity, which capture complementary information from the skeleton joints for encoding them into images. Encoding the skeleton data features into images is an alternative to the traditional video-processing approach and it helps in making the classification task less complex. The Distance and Distance Velocity encoded images have been stacked depth-wise and fed into a Convolutional Neural Network model which is a modified version of Inception-ResNet. Similarly, the Angle and Angle Velocity encoded images have been stacked depth-wise and fed into the same network. After training these models, deep features have been extracted from the pre-final layer of the networks, and the obtained feature representation is optimized by a nature-inspired metaheuristic, called Ant Lion Optimizer, to eliminate the non-informative or misleading features and to reduce the dimensionality of the feature set. DSwarm-Net has been evaluated on three publicly available HAR datasets, namely UTD-MHAD, HDM05, and NTU RGB+D 60 achieving competitive results, thus confirming the superiority of the proposed model compared to state-of-the-art models.

A walk in the black-box: 3D visualization of large neural networks in virtual reality

“…The ability to visualize popular, state-of-the-art architectures raises new questions for future work. With the visualization embedded in a 3D world space it seems convenient to consider machine learning problems with 3D input data such as in pose estimation [38,39]. Applying a 3D visual analysis system would be interesting, because not only the networks, but also the input data can be shown in 3D.…”

An Intelligent Fusion Model with Portfolio Selection and Machine Learning for Stock Market Prediction

“…Prior research has established that machine learning techniques outperform all other predicting stock market directionality [ 50 ]. Traditional models are less flexible than AI approaches [ 51 , 52 ]. Several machine learning algorithms have been investigated in the past [ 53 , 54 ].…”

Multiple spatial residual network for object detection

“…A correlation learning mechanism (CLM) of deep neural network architectures by combining convolutional neural network is proposed [3]. Authors propose DSwarm-Net [2], a framework by employing deep learning and swarm intelligence-based metaheuristic for HAR that uses 3D skeleton data.…”

“…Object detection is an important computer vision task [1][2][3] that deals with detecting instances [4,5] of visual objects of a certain class (such as humans [6], animals or cars [7]) in digital images [8,9]. In the past 2 decades, a variety of deep learning-based object detection algorithms have been proposed [10][11][12].…”