Deep Edge Computing for Videos

Abstract

This paper provides a modular architecture with deep neural networks as a solution for realtime video analytics in an edge-computing environment. The modular architecture consists of two networks of Front-CNN (Convolutional Neural Network) and Back-CNN, where we adopt Shallow 3D CNN (S3D) as the Front-CNN and a pre-trained 2D CNN as the Back-CNN. The S3D (i.e., the Front CNN) is in charge of condensing a sequence of video frames into a feature map with three channels. That is, the S3D takes a set of sequential frames in the video shot as input and yields a learned 3 channel feature map (3CFM) as output. Since the 3CFM is compatible with the three-channel RGB color image format, we can use the output of the S3D (i.e., the 3CFM) as the input to a pre-trained 2D CNN of the Back-CNN for the transfer-learning. This serial connection of Front-CNN and Back-CNN architecture is end-to-end trainable to learn both spatial and temporal information of videos. Experimental results on the public datasets of UCF-Crime and UR-Fall Detection show that the proposed S3D-2DCNN model outperforms the existing methods and achieves state-of-the-art performance. Moreover, since our Front-CNN and Back-CNN modules have a shallow S3D and a light-weighted 2D CNN, respectively, it is suitable for real-time video recognition in edge-computing environments. We have implemented our CNN model on NVIDIA Jetson Nano Developer as an edge-computing device to show its real-time execution.

Audio-Visual Action Recognition Using Transformer Fusion Network

“…In the temporal stream, T frames, each having dimensions of W × H × C, are selected by uniform sampling. Then, these T frames are condensed into a single frame of W × H × C by the S3D (Shallow 3D CNN) motion module [27]. Notably, the S3D module does not utilize fixed weights; instead, its weights are initialized and updated during training, enabling a more adaptive and robust representation of motion features.…”

Anomaly detection using edge computing in video surveillance system: review

“…The aim is to utilize available resources by task partitioning and pre-processing the video data using a DNN model. [ 52 ] proposes a solution for real-time videos by designing a Front-CNN consisting of a Shallow 3D CNN and pre-trained 2D CNN as the Back-CNN. This end-to-end trainable architecture is capable of learning both spatiotemporal information of videos thereby achieving state-of-the-art performance.…”