Online Spatio-temporal 3D Convolutional Neural Network for Early Recognition of Handwritten Gestures

Abstract

Inspired by recent spatio-temporal Convolutional Neural Networks in computer vision field, we propose OLT-C3D (Online Long-Term Convolutional 3D), a new architecture based on a 3D Convolutional Neural Network (3D CNN) to address the complex task of early recognition of 2D handwritten gestures in real time. The input signal of the gesture is translated into an image sequence along time with the trajectory history. The image sequence is passed into our 3D CNN OLT-C3D which gives a prediction at each new frame. OLT-C3D is coupled with an integrated temporal reject system to postpone the decision in time if more information is needed. Moreover our system is end-to-end trainable, OLT-C3D and the temporal reject system are jointly trained to optimize the earliness of the decision. Our approach achieves superior performances on two complementary and freely available datasets: ILGDB and MTGSetB.

Early Recognition of Untrimmed Handwritten Gestures with Spatio-Temporal 3D CNN

“…Our method extends the Online Long-Term Convolutional 3D (OLT-C3D) [7] network to address the early recognition of untrimmed gestures task. First, the online signal of the trace of fingers on the device has to be translated into an image sequence.…”

“…Yamagata et al [9] designed an approach explicitly modeling the trajectory bifurcations between handwritten digits, an LSTM network is used to predict class and the future trajectory. Recently, a new approach based on a 3D Convolutional Neural Network (CNN) called OLT-C3D (for Online Long-Term Convolutional 3D) [7] has been designed to handle long-term visibility without the need of any recurrence layer thanks to temporal dilated convolutions. This approach has a reject system to avoid classification errors in early stages.…”

“…Also, the model is trained with a weakly supervised strategy by predicting future postures, this helps the network to generalize well. Another way to handle early stages is to define a reject option strategy related to the confidence like some other approaches [6], [7], [14].…”

“…Few works address this challenging problem, most of them addressed the early recognition of action made by a full human body, either from RGB videos [1] or 3D skeleton videos [2]- [4]. Regarding 2D gesture recognition, some work addressed the early recognition of trimmed gestures [5]- [7], but not in untrimmed context.…”

“…To address this problem, we present an approach based on a spatio-temporal 3D CNN called OLT-C3D [7] designed for trimmed gesture recognition. OLT-C3D, coupled with a temporal reject system, is able to postpone a detection to avoid misclassified detection, but it can be unstable because the consistence between frames is not explicitly trained, making it unreliable in untrimmed context.…”