A transfer learning-based efficient spatiotemporal human action recognition framework for long and overlapping action classes

Bilal, Muhammad; Maqsood, Muazzam; Yasmin, Sadaf; Hasan, Najam Ul; Rho, Seungmin

doi:10.1007/s11227-021-03957-4

Cited by 28 publications

(12 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Deep learning-based solutions for computer vision have made it easier to deliver technical content for educational purposes or clinical use. It has been mentioned in other papers that conventional image data contain many hidden pieces of information and patterns that can be used for human activity recognition (HAR), and HAR can be applied to many areas such as behavioral analysis, intelligent video surveillance, and robot vision [36]. In terms of erroneous classification, existing hand-engineered and machine learning-based solutions have little or no ability to handle overlapping tasks.…”

Section: Discussionmentioning

confidence: 99%

Deep Learning Approaches to Automated Video Classification of Upper Limb Tension Test

Choi

Heo

2021

Healthcare

View full text Add to dashboard Cite

The purpose of this study was to classify ULTT videos through transfer learning with pre-trained deep learning models and compare the performance of the models. We conducted transfer learning by combining a pre-trained convolution neural network (CNN) model into a Python-produced deep learning process. Videos were processed on YouTube and 103,116 frames converted from video clips were analyzed. In the modeling implementation, the process of importing the required modules, performing the necessary data preprocessing for training, defining the model, compiling, model creation, and model fit were applied in sequence. Comparative models were Xception, InceptionV3, DenseNet201, NASNetMobile, DenseNet121, VGG16, VGG19, and ResNet101, and fine tuning was performed. They were trained in a high-performance computing environment, and validation and loss were measured as comparative indicators of performance. Relatively low validation loss and high validation accuracy were obtained from Xception, InceptionV3, and DenseNet201 models, which is evaluated as an excellent model compared with other models. On the other hand, from VGG16, VGG19, and ResNet101, relatively high validation loss and low validation accuracy were obtained compared with other models. There was a narrow range of difference between the validation accuracy and the validation loss of the Xception, InceptionV3, and DensNet201 models. This study suggests that training applied with transfer learning can classify ULTT videos, and that there is a difference in performance between models.

show abstract

Section: Discussionmentioning

confidence: 99%

Deep Learning Approaches to Automated Video Classification of Upper Limb Tension Test

Choi

Heo

2021

Healthcare

View full text Add to dashboard Cite

show abstract

“…Many researchers now use RNNs and LSTMs to simulate activity changes over time. LRCNs can learn features from variable-length inputs, unlike traditional input methods [25]. Donahue et al [26] use encoder-decoder architecture to extract video representations.…”

Section: Convnets With Rnnsmentioning

confidence: 99%

“…Researchers are struggling to design systems that can apply experience learned from previous tasks to improve performance on a new task. Transfer learning can reduce the architecture learning time and effort, resulting in more robust and useful activity recognition systems [25]. Muhammad et al [23] reveal that transfer learning is a good approach to action recognition.…”

Section: Transfer Learning With Deep Learning As a Feature Extractormentioning

confidence: 99%

Leveraging Transfer Learning for Spatio-Temporal Human Activity Recognition from Video Sequences

Butt¹,

Ullah²,

Letchmunan³

et al. 2023

Computers, Materials &Amp; Continua

View full text Add to dashboard Cite

Human Activity Recognition (HAR) is an active research area due to its applications in pervasive computing, human-computer interaction, artificial intelligence, health care, and social sciences. Moreover, dynamic environments and anthropometric differences between individuals make it harder to recognize actions. This study focused on human activity in video sequences acquired with an RGB camera because of its vast range of real-world applications. It uses two-stream ConvNet to extract spatial and temporal information and proposes a fine-tuned deep neural network. Moreover, the transfer learning paradigm is adopted to extract varied and fixed frames while reusing object identification information. Six state-of-the-art pre-trained models are exploited to find the best model for spatial feature extraction. For temporal sequence, this study uses dense optical flow following the two-stream Con-vNet and Bidirectional Long Short Term Memory (BiLSTM) to capture longterm dependencies. Two state-of-the-art datasets, UCF101 and HMDB51, are used for evaluation purposes. In addition, seven state-of-the-art optimizers are used to fine-tune the proposed network parameters. Furthermore, this study utilizes an ensemble mechanism to aggregate spatial-temporal features using a four-stream Convolutional Neural Network (CNN), where two streams use RGB data. In contrast, the other uses optical flow images. Finally, the proposed ensemble approach using max hard voting outperforms state-ofthe-art methods with 96.30% and 90.07% accuracies on the UCF101 and HMDB51 datasets.

show abstract

“…An approach with a novel temporal-spatial pooling block for action classification, which can learn pool discriminative frames and pixels in a certain clip, has been recently proposed in [ 90 ]. Similarly, in [ 91 ], an efficient spatiotemporal human action recognition framework for long and overlapping action classes has been proposed. Fine-tuned pre-trained CNN models were exploited to learn the spatial relationship at the frame level whereas an optimized Deep Autoencoder [ 92 ] was used to squeeze high-dimensional deep features.…”

Section: Recent Advances In Human Motion Analysismentioning

confidence: 99%

Video-Based Automatic Baby Motion Analysis for Early Neurological Disorder Diagnosis: State of the Art and Future Directions

Leo

Bernava

Carcagnì

et al. 2022

Sensors

View full text Add to dashboard Cite

Neurodevelopmental disorders (NDD) are impairments of the growth and development of the brain and/or central nervous system. In the light of clinical findings on early diagnosis of NDD and prompted by recent advances in hardware and software technologies, several researchers tried to introduce automatic systems to analyse the baby’s movement, even in cribs. Traditional technologies for automatic baby motion analysis leverage contact sensors. Alternatively, remotely acquired video data (e.g., RGB or depth) can be used, with or without active/passive markers positioned on the body. Markerless approaches are easier to set up and maintain (without any human intervention) and they work well on non-collaborative users, making them the most suitable technologies for clinical applications involving children. On the other hand, they require complex computational strategies for extracting knowledge from data, and then, they strongly depend on advances in computer vision and machine learning, which are among the most expanding areas of research. As a consequence, also markerless video-based analysis of movements in children for NDD has been rapidly expanding but, to the best of our knowledge, there is not yet a survey paper providing a broad overview of how recent scientific developments impacted it. This paper tries to fill this gap and it lists specifically designed data acquisition tools and publicly available datasets as well. Besides, it gives a glimpse of the most promising techniques in computer vision, machine learning and pattern recognition which could be profitably exploited for children motion analysis in videos.

show abstract

A transfer learning-based efficient spatiotemporal human action recognition framework for long and overlapping action classes

Cited by 28 publications

References 37 publications

Deep Learning Approaches to Automated Video Classification of Upper Limb Tension Test

Deep Learning Approaches to Automated Video Classification of Upper Limb Tension Test

Leveraging Transfer Learning for Spatio-Temporal Human Activity Recognition from Video Sequences

Video-Based Automatic Baby Motion Analysis for Early Neurological Disorder Diagnosis: State of the Art and Future Directions

Contact Info

Product

Resources

About