Simultaneous Prediction of Wrist and Hand Motions via Wearable Ultrasound Sensing for Natural Control of Hand Prostheses

Abstract

Simultaneous prediction of wrist and hand motions is essential for the natural interaction with hand prostheses. In this paper, we propose a novel multi-out Gaussian process (MOGP) model and a multi-task deep learning (MTDL) algorithm to achieve simultaneous prediction of wrist rotation (pronation/supination) 1 and finger gestures for transradial amputees via a wearable ultrasound array. We target six finger gestures with concurrent wrist rotation in four transradial amputees. Results show that MOGP outperforms previously reported subclass discriminant analysis for both predictions of discrete finger gestures and continuous wrist rotation. Moreover, we find that MTDL has the potential to improve the accuracy of finger gesture prediction compared to MOGP and classification-specific deep learning, albeit at the expense of reducing the accuracy of wrist rotation prediction. Extended comparative analysis shows the superiority of ultrasound over surface electromyography. This paper prioritizes exploring the performance of wearable ultrasound on the simultaneous prediction of wrist and hand motions for transradial amputees, demonstrating the potential of ultrasound in future prosthetic control. Our ultrasound-based adaptive prosthetic control dataset (UltraPro) will be released to promote the development of the prosthetic community.Index Terms-Simultaneous wrist and hand motion control, transradial amputees, wearable ultrasound, multi-out Gaussian process, multi-task deep learning, UltraPro dataset.

Novel near E-Field Topography Sensor for Human–Machine Interfacing in Robotic Applications

“…Comparing this model with others in the literature, the proposed model provides additional complexity by performing regression with a much-higher-dimensional output while maintaining a high level of performance. Although a direct comparison of the performances is difficult given the limited existing data for finger- and thumb-regression performances, the wrist regression demonstrates a similarly high level of performance compared with those of other regressions described in the literature [ 14 , 30 ].The finger regression which had 4 DoF and the worst-performing RMSE of all the regression tasks performed, exceeded the R 2 value claimed by [ 14 ] for the 3-DoF wrist regression, with a value of 0.83. However, this performance is achieved with significantly less pre-processing and manipulation of the underlying sensors’ input data, using only the unmodified digital output from the sensors, windowed into sequences.…”

“…Given the limited number of existing models for pure regression on hand and wrist movements, with some focusing on wrist regression alongside hand movement classification [ 14 ] or simplifying hand movements to the overall opening and closing of the hand [ 30 ], some goals were set additionally to determine the desired model performance. Although the MSE was used as the loss function for the model, the root-mean-square error (RMSE) was used as the performance metric to measure the accuracy of the model.…”

“…The elastic- or pressure-based attachment methods required to achieve consistent contact result in contact effects, which can cause changes in the output of the sensors that are not a result of muscle activation. Another approach that can measure muscle activity through the forearm is ultrasound [ 14 ]. However, this tends to require a large sensor, which can be cumbersome to use and still requires skin contact.…”

“…Some works use these sensors and the HMI differently and use these sensors for human–robot interactions by considering the 3D position of the hand as the output [ 29 ]. Other works seek to address the serial single-degree-of-freedom issue by considering individual degrees of freedom (DoFs) independently through regression for continuous motion [ 14 , 30 ]. Although regression is beginning to see use in joint angle predictions for wrist movements, solutions for the whole hand typically rely on classification or a reduction in the DoFs for the finger movements.…”

A-Mode Ultrasound-Based Prediction of Transfemoral Amputee Prosthesis Walking Kinematics via an Artificial Neural Network

“…Multiple able-bodied studies have applied the technology to gesture recognition [44], force estimation [45], and wrist/hand kinematics estimation [46]. Furthermore, Amode has allowed for finger gesture recognition and wrist rotation estimation with transradial amputee subjects [47], as well as ambulation mode recognition in above-knee amputee subjects [48]. However, it is not known whether ultrasound can be used to predict prosthesis kinematics in above-knee amputees.…”