Modelling a Spatial-Motion Deep Learning Framework to Classify Dynamic Patterns of Videos

Abstract

Video classification is an essential process for analyzing the pervasive semantic information of video content in computer vision. Traditional hand-crafted features are insufficient when classifying complex video information due to the similarity of visual contents with different illumination conditions. Prior studies of video classifications focused on the relationship between the standalone streams themselves. In this paper, by leveraging the effects of deep learning methodologies, we propose a two-stream neural network concept, named state-exchanging long short-term memory (SE-LSTM). With the model of spatial motion state-exchanging, the SE-LSTM can classify dynamic patterns of videos using appearance and motion features. The SE-LSTM extends the general purpose of LSTM by exchanging the information with previous cell states of both appearance and motion stream. We propose a novel two-stream model Dual-CNNSELSTM utilizing the SE-LSTM concept combined with a Convolutional Neural Network, and use various video datasets to validate the proposed architecture. The experimental results demonstrate that the performance of the proposed two-stream Dual-CNNSELSTM architecture significantly outperforms other datasets, achieving accuracies of 81.62%, 79.87%, and 69.86% with hand gestures, fireworks displays, and HMDB51 datasets, respectively. Furthermore, the overall results signify that the proposed model is most suited to static background dynamic patterns classifications.

Tackling Explicit Material from Online Video Conferencing Software for Education Using Deep Attention Neural Architectures

“…Arachchi et al [ 12 ] introduced a state-exchanging long short-term memory (SE-LSTM) two-stream neural network approach, based on the benefits of using spatial and motion information to identify dynamic patterns. This method was used to identify movie reactions using appearance motion characteristics.…”

Development of typical "state" software patterns for Cortex­M microcontrollers in real time

“…The software that makes it possible to reach high speeds for image processing is described in [9]. The authors do not address the mathematical notation and software for formalizing and switching the RTTS states within the State pattern.…”

“…A variant of solving the tasks related to the software of the ASC prototype, outlined in [9][10][11][12][13][14][15][16][17], concerning the issues of increasing the software speed, unification, maintenance, and mathematical modeling, is the development of typical software State templates (patterns) for the Cortex-M architecture MCs. These patterns can be implemented in the Polled Loop System variant or through the microcontroller Interrupt Driven system.…”

Video Description Model Based on Temporal-Spatial and Channel Multi-Attention Mechanisms

“…Video description is widely used in advanced intelligent technology, including smart city, smart transportation and smart home [1][2][3][4][5]. Video description technology is a part of computer vision and natural language processing, which has attracted much attention in recent years [6][7][8][9].…”