3DFCNN: real-time action recognition using 3D deep neural networks with raw depth information

Sánchez-Caballero, A.; Diz, Sergio; Fuentes-Jiménez, David; Losada, Cristina; Marrón, M.; Casillas-Pérez, David; Sarker, Mohammad Ibrahim

doi:10.1007/s11042-022-12091-z

Cited by 41 publications

(20 citation statements)

References 94 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Action detection, often the main task of video understanding, typically estimates a 2D bounding box (region in the image) and label for each action event on video frames. Convolution neural networks (CNNs) have been successfully applied to vehicle motion detection, as in [26][27][28][29][30]. With respect to autonomous driving, works using ego-vehicle cameras such as [31][32][33][34] focus on detecting vehicle motion.…”

Section: Video Understandingmentioning

confidence: 99%

Real-Time Idling Vehicles Detection Using Combined Audio-Visual Deep Learning

Li,

Mangin,

Saha

et al. 2024

Advances in Transdisciplinary Engineering

View full text Add to dashboard Cite

Combustion vehicle emissions contribute to poor air quality and release greenhouse gases into the atmosphere, and vehicle pollution has been associated with numerous adverse health effects. Roadways with extensive waiting and/or passenger drop-off, such as schools and hospital drop-off zones, can result in a high incidence and density of idling vehicles. This can produce micro-climates of increased vehicle pollution. Thus, the detection of idling vehicles can be helpful in monitoring and responding to unnecessary idling and be integrated into real-time or off-line systems to address the resulting pollution. In this paper, we present a real-time, dynamic vehicle idling detection algorithm. The proposed idle detection algorithm and notification rely on an algorithm to detect these idling vehicles. The proposed method relies on a multisensor, audio-visual, machine-learning workflow to detect idling vehicles visually under three conditions: moving, static with the engine on, and static with the engine off. The visual vehicle motion detector is built in the first stage, and then a contrastive-learning-based latent space is trained for classifying static vehicle engine sound. We test our system in real-time at a hospital drop-off point in Salt Lake City. This in situ dataset was collected and annotated, and it includes vehicles of varying models and types. The experiments show that the method can detect engine switching on or off instantly and achieves 71.02 average precision (AP) for idle detection and 91.06 for engine off detection.

show abstract

Section: Video Understandingmentioning

confidence: 99%

Real-Time Idling Vehicles Detection Using Combined Audio-Visual Deep Learning

Li,

Mangin,

Saha

et al. 2024

Advances in Transdisciplinary Engineering

View full text Add to dashboard Cite

show abstract

“…Action Recognition from 3D Data. 3D action recognition models can be broadly categorized in to depthbased, [27][28][29][30][31] skeleton-based, [32][33][34][35][36][37][38] and point-cloud-based. [39][40][41][42] Depth representations provide reliable 3D geometric cues which can be robust to different viewpoints.…”

Section: Related Workmentioning

confidence: 99%

Leveraging body pose estimation for gesture recognition in human-robot interaction using synthetic data

Zhu

Melo

Hauptmann

2023

Synthetic Data for Artificial Intelligence and Machine Learning: Tools, Techniques, and Applications

View full text Add to dashboard Cite

Effectively recognizing human gestures from variant viewpoints plays a fundamental role in the successful collaboration between humans and robots. Deep learning approaches have achieved promising performance in gesture recognition. However, they are usually data-hungry and require large-scale labeled data, which are not usually accessible in a practical setting. Synthetic data, on the other hand, can be easily obtained from simulators with fine-grained annotations and variant modalities. Existing state-of-the-art approaches have shown promising results using synthetic data, but there is still a large performance gap between the models trained on synthetic data and real data. To learn domain-invariant feature representations, we propose a novel approach which jointly takes RGB videos and 3D meshes as inputs to perform robust action recognition. We empirically validate our model on the RoCoG-v2 dataset, which consists of a variety of real and synthetic videos of gestures from the ground and air perspectives. We show that our model trained on synthetic data can outperform state-of-the-art models under the same training setting and models trained on real data.

show abstract

“…As in the case of RGB-based HAR works, the first depth-based studies employed methods based on handcrafted descriptors [28,41,65,67], but eventually works using deep learning became the main approach [47]. Besides, DNN-based approaches have been proved [69] to be more robust and suitable for challenging large datasets than handcrafted features-based methods, but with much higher computational costs, which makes it difficult to use them in real-time applications.…”

Section: Introductionmentioning

confidence: 99%

“…The use of DNNs for HAR requires encoding the spatio-temporal information [7]. To do that, there are several approaches that create ad-hoc representations such as depth motion maps [73] or dynamic images [72,74,76,79,81], whereas other works use specific DNNs, such as the 3D convolutional neural networks (3DCNNs) [36,47,54,80] or the recurrent neural networks (RNN) [11,31,49,50], that processes video-sequences. A particular RNN which solves the exploding or vanishing gradient problem is the long short-term memory (LSTM) [17], which can successfully learn patterns in long sequences like videos by stacking several layers.…”

Section: Introductionmentioning

confidence: 99%

Real-time human action recognition using raw depth video-based recurrent neural networks

Sánchez-Caballero

Fuentes-Jiménez

Losada

2022

Multimed Tools Appl

Self Cite

View full text Add to dashboard Cite

This work proposes and compare two different approaches for real-time human action recognition (HAR) from raw depth video sequences. Both proposals are based on the convolutional long short-term memory unit, namely ConvLSTM, with differences in the architecture and the long-term learning. The former uses a video-length adaptive input data generator (stateless) whereas the latter explores the stateful ability of general recurrent neural networks but is applied in the particular case of HAR. This stateful property allows the model to accumulate discriminative patterns from previous frames without compromising computer memory. Furthermore, since the proposal uses only depth information, HAR is carried out preserving the privacy of people in the scene, since their identities can not be recognized. Both neural networks have been trained and tested using the large-scale NTU RGB+D dataset. Experimental results show that the proposed models achieve competitive recognition accuracies with lower computational cost compared with state-of-the-art methods and prove that, in the particular case of videos, the rarely-used stateful mode of recurrent neural networks significantly improves the accuracy obtained with the standard mode. The recognition accuracies obtained are 75.26% (CS) and 75.45% (CV) for the stateless model, with an average time consumption per video of 0.21 s, and 80.43% (CS) and 79.91%(CV) with 0.89 s for the stateful one.

show abstract

3DFCNN: real-time action recognition using 3D deep neural networks with raw depth information

Cited by 41 publications

References 94 publications

Real-Time Idling Vehicles Detection Using Combined Audio-Visual Deep Learning

Real-Time Idling Vehicles Detection Using Combined Audio-Visual Deep Learning

Leveraging body pose estimation for gesture recognition in human-robot interaction using synthetic data

Real-time human action recognition using raw depth video-based recurrent neural networks

Contact Info

Product

Resources

About