Two Stage Continuous Gesture Recognition Based on Deep Learning

Abstract

The paper proposes an effective continuous gesture recognition method, which includes two modules: segmentation and recognition. In the segmentation module, the video frames are divided into gesture frames and transitional frames by using the information of hand motion and appearance, and continuous gesture sequences are segmented into isolated sequences. In the recognition module, our method exploits the spatiotemporal information embedded in RGB and depth sequences. For the RGB modality, our method adopts Convolutional Long Short-Term Memory Networks to learn long-term spatiotemporal features from short-term spatiotemporal features obtained from a 3D convolutional neural network. For the depth modality, our method converts a sequence into Dynamic Images and Motion Dynamic Images through weighted rank pooling and feed them into Convolutional Neural Networks, respectively. Our method has been evaluated on both ChaLearn LAP Large-scale Continuous Gesture Dataset and Montalbano Gesture Dataset and achieved state-of-the-art performance.

A 3DCNN-LSTM Multi-Class Temporal Segmentation for Hand Gesture Recognition

“…Then, to increase performance, an additional convolutional layer and an increased sample size were considered. The first two tests were evaluated over mini-batches of 13 epochs, following the segmentation classifier proposed by Wang [28]. The last two tests were evaluated over a batch size of 64 epochs, a training batch size also presented in Wang's investigation [28].…”

“…The first two tests were evaluated over mini-batches of 13 epochs, following the segmentation classifier proposed by Wang [28]. The last two tests were evaluated over a batch size of 64 epochs, a training batch size also presented in Wang's investigation [28]. A 12GB NVIDIA Tesla K80 graphics processing unit provided by Google Colaboratory was used for training of the 20BN Jester dataset for the baseline model, TensorFlow [35] was used to deploy the model, and the training took approximately nine and a half hours.…”

“…The ability to automatically detect action in video recordings has an essential function for different applications that require end-to-end process automation. But, while much work has been produced on increasing the accuracy of hand gesture recognition models and enhancing the strength of these approaches [3], [5], [26], just a few attempts have been presented for temporal segmentation [27], [28].…”

“…However, the length of the sliding volume was fixed, often cutting part of the critical features of the gestures. Moreover, appearance and hand motion information complement a temporal segmentation classifier [28]. Still, Camgoz et al also used only time-series data detected from hand motion, with no appearance information [31].…”

“…Still, Camgoz et al also used only time-series data detected from hand motion, with no appearance information [31]. In contrast, Wang presented a segmentation method that contained both action and appearance-based information, and used both RGB and depth capture modalities [28].…”