A Hybrid Multimodal Fusion Framework for sEMG-ACC-Based Hand Gesture Recognition

Duan, Shengcai; Wu, Le; Xue, Bo; Liu, Aiping; Qian, Ruobing; Chen, Xun

doi:10.1109/jsen.2022.3231925

Cited by 14 publications

(17 citation statements)

References 38 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As a result, we adopted the data splitting methodology recommended in the 'Evaluation Metric' section for subsequent evaluations. This method ensures a fair comparison with most recent DL models developed using Ninapro DB2 data [6,10,23,30].…”

Section: Experimental Settingmentioning

confidence: 99%

“…Recent studies have utilized DL models to fuse sEMG and ACC signals from sparse multi-channel sensors for improved HGR. The majority of these approaches combine traditional feature extraction with DNNs, with subject-specific assessments [10,30,49,50]. Notably, state-of-the-art HGR methods using end-toend sEMG and ACC DL models, such as NIDA-TCN [31] and two-stream LSTM-Res [35], lack fair comparative evaluations on public databases like Ninapro.…”

Section: Models Comparementioning

confidence: 99%

“…To ensure equitable comparisons, we selected a 400 ms window in this study. We also listed comparisons between NIMFT and non-end-to-end HGR methods reported in the existing literature [10,30,49,50] in table 3. In conclusion, our overall recognition accuracy achieved 91.92%, establishing NIMFT as the top-performing end-to-end model in terms of recognition accuracy when employing inter-subject training and evaluation.…”

Section: Models Comparementioning

confidence: 99%

“…However, feature engineering for HGR using ACC signals is still underdeveloped, lacking consensus on the best ACC features. Some studies adopt similar feature extraction methods as used for sEMG signals [30], while others rely on basic features such as mean, maximum, and minimum values [31]. An endto-end deep learning model that fuses sEMG and ACC modalities enables automatic feature learning without redundant manual crafting.…”

Section: Introductionmentioning

confidence: 99%

“…The fusion of sEMG and ACC modalities in prior studies often involved direct fusion methods such as summation or concatenation, known as the invasive method. However, this approach lacks multilevel intermodal interactive knowledge as it performs fusion solely in either the low-level data space or high-level decision space [30]. Moreover, the inherent heterogeneity of these signals may result in disruptions of the original signal characteristics during invasive fusion.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Transferable non-invasive modal fusion-transformer (NIMFT) for end-to-end hand gesture recognition

Xu,

Zhao,

et al. 2024

J. Neural Eng.

View full text Add to dashboard Cite

Objective. Recent studies have shown that integrating IMU signals with surface electromyographic (sEMG) can greatly improve hand gesture recognition (HGR) performance in applications such as prosthetic control and rehabilitation training. However, current deep learning models for multimodal HGR encounter difficulties in invasive modal fusion, complex feature extraction from heterogeneous signals, and limited inter-subject model generalization. To address these challenges, this study aims to develop an end-to-end and inter-subject transferable model that utilizes non-invasively fused sEMG and acceleration (ACC) data. Approach. The proposed NIMFT model utilizes 1D-CNN-based patch embedding for local information extraction and employs a multi-head cross-attention (MCA) mechanism to non-invasively integrate sEMG and ACC signals, stabilizing the variability induced by sEMG. The proposed architecture undergoes detailed ablation studies after hyperparameter tuning. Transfer learning is employed by fine-tuning a pre-trained model on new subject and a comparative analysis is performed between the fine-tuning and subject-specific model. Additionally, the performance of NIMFT is compared to state-of-the-art fusion models. Main results. The NIMFT model achieved recognition accuracies of 93.91%, 91.02%, and 95.56% on the three action sets in the Ninapro DB2 dataset. The proposed embedding method and MCA outperformed the traditional invasive modal fusion transformer by 2.01% (embedding) and 1.23% (fusion), respectively. In comparison to subject-specific models, the fine-tuning model exhibited the highest average accuracy improvement of 2.26%, achieving a final accuracy of 96.13%. Moreover, the NIMFT model demonstrated superiority in terms of accuracy, recall, precision, and F1-score compared to the latest modal fusion models with similar model scale. Significance. The NIMFT is a novel end-to-end HGR model, utilizes a non-invasive MCA mechanism to integrate long-range intermodal information effectively. Compared to recent modal fusion models, it demonstrates superior performance in inter-subject experiments and offers higher training efficiency and accuracy levels through transfer learning than subject-specific approaches.

show abstract

Section: Experimental Settingmentioning

confidence: 99%

Section: Models Comparementioning

confidence: 99%

Section: Models Comparementioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Transferable non-invasive modal fusion-transformer (NIMFT) for end-to-end hand gesture recognition

Xu,

Zhao,

et al. 2024

J. Neural Eng.

View full text Add to dashboard Cite

show abstract

Exploring the Potential of Machine Learning and Deep Learning for COVID‐19 Detection

Bashir,

Firdous,

Rufai

2024

Deep Learning Techniques for Automation and Industrial Applications

View full text Add to dashboard Cite

Wearable Sensor‐Based Multi‐modal Fusion Network for Automated Gait Dysfunction Assessment in Children with Cerebral Palsy

Tang,

Wang,

Lian

et al. 2024

Advanced Intelligent Systems

View full text Add to dashboard Cite

Gait, fundamental to human movement, becomes compromised in cerebral palsy (CP), a childhood‐onset central nervous system motor disorder. Precise assessment of patients’ gait is crucial for tailored rehabilitation interventions. Currently, clinical scales assessing CP gait dysfunction mostly, while valuable, rely on subjective clinician observations. To enhance objectivity and efficiency in CP diagnosis and rehabilitation, there is a need for more objective assessment procedures. This study introduces a multi‐modal and multi‐scale feature fusion (MMFF) framework, a new framework for automating gait dysfunction assessment in children with CP. By utilizing surface electromyography and acceleration signals recorded during children's walking, MMFF generates a feature vector enriched with adaptively refined feature maps, cross‐mode correlations, and both local and global information. Validation of MMFF's effectiveness is evident through an accomplished classification accuracy of 99.13%. The mean values for precision, recall, and F1‐score in Gross Motor Function Classification System (GMFCS)‐1, GMFCS‐2, and GMFCS‐3, reaching 99.00%, 99.00%, and 98.33%, respectively, further reflect the accuracy of functional assessments at each level. This study underscores MMFF's potential as an objective, streamlined tool for clinicians, promising improved gait assessment and well‐informed rehabilitation strategies for children with CP.

show abstract

A Hybrid Multimodal Fusion Framework for sEMG-ACC-Based Hand Gesture Recognition

Cited by 14 publications

References 38 publications

Transferable non-invasive modal fusion-transformer (NIMFT) for end-to-end hand gesture recognition

Transferable non-invasive modal fusion-transformer (NIMFT) for end-to-end hand gesture recognition

Exploring the Potential of Machine Learning and Deep Learning for COVID‐19 Detection

Wearable Sensor‐Based Multi‐modal Fusion Network for Automated Gait Dysfunction Assessment in Children with Cerebral Palsy

Contact Info

Product

Resources

About