Real-Time Analysis of Hand Gesture Recognition with Temporal Convolutional Networks

Abstract

In recent years, the successful application of Deep Learning methods to classification problems has had a huge impact in many domains. (1) Background: In biomedical engineering, the problem of gesture recognition based on electromyography is often addressed as an image classification problem using Convolutional Neural Networks. Recently, a specific class of these models called Temporal Convolutional Networks (TCNs) has been successfully applied to this task. (2) Methods: In this paper, we approach electromyography-based hand gesture recognition as a sequence classification problem using TCNs. Specifically, we investigate the real-time behavior of our previous TCN model by performing a simulation experiment on a recorded sEMG dataset. (3) Results: The proposed network trained with data augmentation yields a small improvement in accuracy compared to our existing model. However, the classification accuracy is decreased in the real-time evaluation, showing that the proposed TCN architecture is not suitable for such applications. (4) Conclusions: The real-time analysis helps in understanding the limitations of the model and exploring new ways to improve its performance.

Deep Learning for Enhanced Prosthetic Control: Real-Time Motor Intent Decoding for Simultaneous Control of Artificial Limbs

“…Several recent studies have shown that CNNs can effectively extract underlying motor control information from EMG signals in both offline data sets [12]- [14] and real-time control scenarios [15]. Combining CNNs feature-extracting architectures with other network modules (e.g., Recurrent Neural Networks (RNN) [16], Long Short-Term Memory networks (LSTM) [17], [18], or Temporal Convolutional Networks (TCN) [19], [20]) has resulted in even higher offline movement decoding performance.…”

“…A CNN-SE architecture was selected because it demonstrated significant improvement over several of the state-of-the-art CNNs [22] and promising results in bio-signal classification tasks (i.e., electrocardiogram [23], [24] and electroencephalogram [25], [26]), and recently also for offline EMG gesture recognition [27], [28]. The choice of the other three networks allowed us to investigate if the promising offline results of increasing the number of hidden layers in FFNNs [8] and the reportedly high performance of TCN's [19], [20] would translate to an online control scenario.…”

Deep learning for enhanced prosthetic control: Real-time motor intent decoding for simultaneous control of artificial limbs

“…Several recent studies have shown that CNNs can effectively extract underlying motor control information from EMG signals in both offline data sets [12]- [14] and real-time control scenarios [15]. Combining CNNs feature-extracting structures with other network modules (e.g., Recurrent Neural Networks (RNN) [16], Long Short-Term Memory networks (LSTM) [17], [18], or Temporal Convolutional Networks (TCN) [19], [20]) has resulted in even higher offline movement decoding performance.…”

“…A CNN-SE architecture was selected because it demonstrated significant improvement over several of the state-of-the-art CNNs [22] and promising results in bio-signal classification tasks (i.e., electrocardiogram [23], [24] and electroencephalogram [25], [26]), and recently also for offline EMG gesture recognition [27], [28]. The choice of the other three networks allowed us to investigate if the promising offline results of increasing the number of hidden layers in FFNNs [8] and the reportedly high performance of TCN's [19], [20] would translate to real-time control scenario.…”

On the Distribution of Muscle Signals: A Method for Distance-Based Classification of Human Gestures

“…Surface-EMG-based methods for gesture recognition have already been proposed by several publications, among them artificial neural networks, to achieve up to 87% accuracy with a single-channel-sensor when attempting to distinguish four different gestures [5]. Generally, black-box neural networks, particularly convolutional neural networks, have emerged in popularity to analyze sEMG gesture data [6][7][8][9][10][11][12][13][14][15][16][17]. A composition consisting of a Bayes classifier and a k-Nearest-Neighbors classifier can achieve up to 93% accuracy for four different gestures with one channel [18].…”