ViT-HGR: Vision Transformer-based Hand Gesture Recognition from High Density Surface EMG Signals

Montazerin, Mansooreh; Zabihi, Soheil; Rahimian, Elahe; Mohammadi, Arash; Naderkhani, Farnoosh

doi:10.1109/embc48229.2022.9871489

Cited by 28 publications

(14 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This problem can only be solved by allowing the user to directly control the closing speed or by allowing so many speeds to be selected that the differences are effectively negligible. We consider the latter to be unfeasible, especially because the maximum of poses (as far as we are aware) that can be encoded with ≥ 90% accuracy is 65 [9] if we count isometric movements, or 52 [32] without isometric tasks.…”

Section: Discussionmentioning

confidence: 99%

Sensing the Full Dynamics of the Human Hand with a Neural Interface and Deep Learning

Sîmpetru

Arkudas

Braun

et al. 2022

Preprint

View full text Add to dashboard Cite

Theories on the neural control of movement are largely based on movement-sensing devices that capture the dynamics of predefined anatomical landmarks. Neuromuscular interfaces, such as surface electromyography (sEMG), can in theory surpass the limitations imposed by motion-based technologies by sensing the motor commands transmitted by the final pathway of movement, the motor units. The recording of motor unit activity may allow the prediction of the kinetics and kinematics continuously in time and space, without being limited to several biological and physical boundaries that digital cameras or inertial sensors suffer. However, current sEMG decoding algorithms can only predict few degrees of freedom (<3). By combining markerless machine vision and high-density sEMG electrodes, we aimed to test the hypothesis that a physiologically inspired deep neural network can reconstruct the movement of the human hand as precise as digital cameras and with the additional benefit of predicting the underlying forces (i.e., grasping a cup of coffee). We demonstrate that our deep learning model can continuously predict all degrees of freedom of the hand with negligible errors during natural motion tasks from 320 sEMG sensors placed only on the extrinsic hand muscles. Our deep learning model was able to display the 3D hand kinematics and the full force range of the hand digits during isometric contractions. The current results demonstrate that deep learning applied to EMG signals gives access to an unprecedented representation of the final neural code of movement.

show abstract

Section: Discussionmentioning

confidence: 99%

Sensing the Full Dynamics of the Human Hand with a Neural Interface and Deep Learning

Sîmpetru

Arkudas

Braun

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The original ViT models substitute the inherent inductive bias favoring local processing in convolutions with global processing facilitated by multi-head self-attention, resulting in enhanced performance in vision tasks. Recently, Montazerin introduced the ViT-HGR framework, leveraging the ViT architecture to efficiently classify 65 hand gestures using HD-sEMG data with a significant accuracy of 84.6% [51], demonstrating its effectiveness in processing sequential signals like sEMG [51]. However, it is important to note that the patchification using linear projection in the original ViT may lead to the loss of neighborhood dependencies within patches, particularly in closely related sequential signals.…”

Section: Patch Embedding With 1d-cnnmentioning

confidence: 99%

Transferable non-invasive modal fusion-transformer (NIMFT) for end-to-end hand gesture recognition

Xu,

Zhao,

et al. 2024

J. Neural Eng.

View full text Add to dashboard Cite

Objective. Recent studies have shown that integrating IMU signals with surface electromyographic (sEMG) can greatly improve hand gesture recognition (HGR) performance in applications such as prosthetic control and rehabilitation training. However, current deep learning models for multimodal HGR encounter difficulties in invasive modal fusion, complex feature extraction from heterogeneous signals, and limited inter-subject model generalization. To address these challenges, this study aims to develop an end-to-end and inter-subject transferable model that utilizes non-invasively fused sEMG and acceleration (ACC) data. Approach. The proposed NIMFT model utilizes 1D-CNN-based patch embedding for local information extraction and employs a multi-head cross-attention (MCA) mechanism to non-invasively integrate sEMG and ACC signals, stabilizing the variability induced by sEMG. The proposed architecture undergoes detailed ablation studies after hyperparameter tuning. Transfer learning is employed by fine-tuning a pre-trained model on new subject and a comparative analysis is performed between the fine-tuning and subject-specific model. Additionally, the performance of NIMFT is compared to state-of-the-art fusion models. Main results. The NIMFT model achieved recognition accuracies of 93.91%, 91.02%, and 95.56% on the three action sets in the Ninapro DB2 dataset. The proposed embedding method and MCA outperformed the traditional invasive modal fusion transformer by 2.01% (embedding) and 1.23% (fusion), respectively. In comparison to subject-specific models, the fine-tuning model exhibited the highest average accuracy improvement of 2.26%, achieving a final accuracy of 96.13%. Moreover, the NIMFT model demonstrated superiority in terms of accuracy, recall, precision, and F1-score compared to the latest modal fusion models with similar model scale. Significance. The NIMFT is a novel end-to-end HGR model, utilizes a non-invasive MCA mechanism to integrate long-range intermodal information effectively. Compared to recent modal fusion models, it demonstrates superior performance in inter-subject experiments and offers higher training efficiency and accuracy levels through transfer learning than subject-specific approaches.

show abstract

“… Burrello et al (2022) proposed using Bioformers, a supersmall attention-based neural network architecture, to solve the problems of big resource consumption and difficult improvement of accuracy in sEMG signal gesture recognition tasks. Montazerin et al (2022) proposed a gesture recognition model called ViT-HGR, which is the first to introduce visual transformer architecture into high-density sEMG signal gesture recognition tasks, leveraging its parallel computing advantages to overcome the long training time and memory limitations of deep learning models. Liu et al (2023) proposed a CNN-Transformer hybrid network for high-accuracy dynamic gesture recognition.…”

Section: Introductionmentioning

confidence: 99%

A rotary transformer cross-subject model for continuous estimation of finger joints kinematics and a transfer learning approach for new subjects

Lin,

2024

Front. Neurosci.

View full text Add to dashboard Cite

IntroductionSurface Electromyographic (sEMG) signals are widely utilized for estimating finger kinematics continuously in human-machine interfaces (HMI), and deep learning approaches are crucial in constructing the models. At present, most models are extracted on specific subjects and do not have cross-subject generalizability. Considering the erratic nature of sEMG signals, a model trained on a specific subject cannot be directly applied to other subjects. Therefore, in this study, we proposed a cross-subject model based on the Rotary Transformer (RoFormer) to extract features of multiple subjects for continuous estimation kinematics and extend it to new subjects by adversarial transfer learning (ATL) approach.MethodsWe utilized the new subject’s training data and an ATL approach to calibrate the cross-subject model. To improve the performance of the classic transformer network, we compare the impact of different position embeddings on model performance, including learnable absolute position embedding, Sinusoidal absolute position embedding, and Rotary Position Embedding (RoPE), and eventually selected RoPE. We conducted experiments on 10 randomly selected subjects from the NinaproDB2 dataset, using Pearson correlation coefficient (CC), normalized root mean square error (NRMSE), and coefficient of determination (R2) as performance metrics.ResultsThe proposed model was compared with four other models including LSTM, TCN, Transformer, and CNN-Attention. The results demonstrated that both in cross-subject and subject-specific cases the performance of RoFormer was significantly better than the other four models. Additionally, the ATL approach improves the generalization performance of the cross-subject model better than the fine-tuning (FT) transfer learning approach.DiscussionThe findings indicate that the proposed RoFormer-based method with an ATL approach has the potential for practical applications in robot hand control and other HMI settings. The model’s superior performance suggests its suitability for continuous estimation of finger kinematics across different subjects, addressing the limitations of subject-specific models.

show abstract

ViT-HGR: Vision Transformer-based Hand Gesture Recognition from High Density Surface EMG Signals

Cited by 28 publications

References 20 publications

Sensing the Full Dynamics of the Human Hand with a Neural Interface and Deep Learning

Sensing the Full Dynamics of the Human Hand with a Neural Interface and Deep Learning

Transferable non-invasive modal fusion-transformer (NIMFT) for end-to-end hand gesture recognition

A rotary transformer cross-subject model for continuous estimation of finger joints kinematics and a transfer learning approach for new subjects

Contact Info

Product

Resources

About