Improving Robotic Hand Prosthesis Control With Eye Tracking and Computer Vision: A Multimodal Approach Based on the Visuomotor Behavior of Grasping

Cognolato, Matteo; Atzori, Manfredo; Gassert, Roger; Müller, Henning

doi:10.3389/frai.2021.744476

Cited by 17 publications

(16 citation statements)

References 48 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Various possible factors responsible for the poor long-term robustness of HGR have been studied and could be summarised as muscle fatigue, skin conductivity, limb position, electrode displacement, and signal variation over days [10], [11], [12], [13], [14]. To address these issues, we believe that rejecting uncertain predictions [15], [16], [17] is the most straightforward approach compared to others such as multimodal approaches [6], [18], [19], adaptive learning [20], [21], [22], [23], [24], and alternative training protocols [24], [25], [26], [27].…”

Section: Introductionmentioning

confidence: 99%

Robust Long-Term Hand Grasp Recognition With Raw Electromyographic Signals Using Multidimensional Uncertainty-Aware Models

Lin

Palaniappan

Wilde

et al. 2023

IEEE Trans. Neural Syst. Rehabil. Eng.

View full text Add to dashboard Cite

Hand grasp recognition with surface electromyography (sEMG) has been used as a possible natural strategy to control hand prosthetics. However, effectively performing activities of daily living for users relies significantly on the long-term robustness of such recognition, which is still a challenging task due to confused classes and several other variabilities. We hypothesise that this challenge can be addressed by introducing uncertainty-aware models because the rejection of uncertain movements has previously been demonstrated to improve the reliability of sEMG-based hand gesture recognition. With a particular focus on a very challenging benchmark dataset (NinaPro Database 6), we propose a novel end-to-end uncertainty-aware model, an evidential convolutional neural network (ECNN), which can generate multidimensional uncertainties, including vacuity and dissonance, for robust long-term hand grasp recognition. To avoid heuristically determining the optimal rejection threshold, we examine the performance of misclassification detection in the validation set. Extensive comparisons of accuracy under the non-rejection and rejection scheme are conducted when classifying 8 hand grasps (including rest) over 8 subjects across proposed models. The proposed ECNN is shown to improve recognition performance, achieving an accuracy of 51.44% without the rejection option and 83.51% under the rejection scheme with multidimensional uncertainties, significantly improving the current state-of-the-art (SoA) by 3.71% and 13.88%, respectively. Furthermore, its overall rejection-capable recognition accuracy remains stable with only a small accuracy degradation after the last data acquisition over 3 days. These results show the potential design of a reliable classifier that yields accurate and robust recognition performance.

show abstract

Section: Introductionmentioning

confidence: 99%

Robust Long-Term Hand Grasp Recognition With Raw Electromyographic Signals Using Multidimensional Uncertainty-Aware Models

Lin

Palaniappan

Wilde

et al. 2023

IEEE Trans. Neural Syst. Rehabil. Eng.

View full text Add to dashboard Cite

show abstract

“…7 Using the dataset for offline testing, they proposed a multimodal method (first-person image and EMG signals) to detect the grasp types. 8 This method used the gaze point to decide the target object according to the pixel distance between the object mask and the gaze point. Similarly, some studies used eye movement signals to judge which is the target object after detecting multiple objects from the image.…”

Section: Introduction 1| Related Backgroundmentioning

confidence: 99%

“…create a multimodal dataset for intelligent prosthetics, including gaze, visual, myoelectric, and inertial data of grasps 7 . Using the dataset for offline testing, they proposed a multimodal method (first‐person image and EMG signals) to detect the grasp types 8 . This method used the gaze point to decide the target object according to the pixel distance between the object mask and the gaze point.…”

Section: Introductionmentioning

confidence: 99%

i‐MYO: A multi‐grasp prosthetic hand control system based on gaze movements, augmented reality, and myoelectric signals

Shi,

Zhao,

Yang

et al. 2023

Robotics Computer Surgery

View full text Add to dashboard Cite

BackgroundControlling a multi‐grasp prosthetic hand still remains a challenge. This study explores the influence of merging gaze movements and augmented reality in bionics on improving prosthetic hand control.MethodsA control system based on gaze movements, augmented reality, and myoelectric signals (i‐MYO) was proposed. In the i‐MYO, the GazeButton was introduced into the controller to detect the grasp‐type intention from the eye‐tracking signals, and the proportional velocity scheme based on the i‐MYO was used to control hand movement.ResultsThe able‐bodied subjects with no prior training successfully transferred objects in 91.6% of the cases and switched the optimal grasp types in 97.5%. The patient could successfully trigger the EMG to control the hand holding the objects in 98.7% of trials in around 3.2 s and spend around 1.3 s switching the optimal grasp types in 99.2% of trials.ConclusionsMerging gaze movements and augmented reality in bionics can widen the control bandwidth of prosthetic hand. With the help of i‐MYO, the subjects can control a prosthetic hand using six grasp types if they can manipulate two muscle signals and gaze movement.

show abstract

“…All these aspects concur to identify the grasp planned by the user, which determines the correct hand pre-shape, wrist orientation and hand closure, that would be otherwise unknown to the control system. Interestingly, it has been shown that the inclusion of visual information significantly increases the average grasp type classification accuracy [9]. Specifically, in shared autonomy, the hand aperture and wrist orientation can be controlled automatically, based on visual input, while the closure of the fingers can be left to the user.…”

Section: Introductionmentioning

confidence: 99%

Grasp Pre-shape Selection by Synthetic Training: Eye-in-hand Shared Control on the Hannes Prosthesis

Federico¹,

Maiettini²,

Pasquale³

et al. 2022

Preprint

View full text Add to dashboard Cite

We consider the task of object grasping with a prosthetic hand capable of multiple grasp types. In this setting, communicating the intended grasp type often requires a high user cognitive load which can be reduced adopting shared autonomy frameworks. Among these, so-called eye-inhand systems automatically control the hand aperture and preshaping before the grasp, based on visual input coming from a camera on the wrist. In this work, we present an eye-inhand learning-based approach for hand pre-shape classification from RGB sequences. In order to reduce the need for tedious data collection sessions for training the system, we devise a pipeline for rendering synthetic visual sequences of hand trajectories for the purpose. We tackle the peculiarity of the eye-in-hand setting by means of a model for the human arm trajectories, with domain randomization over relevant visual elements. We develop a sensorized setup to acquire real human grasping sequences for benchmarking and show that, compared on practical use cases, models trained with our synthetic dataset achieve better generalization performance than models trained on real data. We finally integrate our model on the Hannes prosthetic hand and show its practical effectiveness. Our code, real and synthetic datasets will be released upon acceptance.

show abstract

Improving Robotic Hand Prosthesis Control With Eye Tracking and Computer Vision: A Multimodal Approach Based on the Visuomotor Behavior of Grasping

Cited by 17 publications

References 48 publications

Robust Long-Term Hand Grasp Recognition With Raw Electromyographic Signals Using Multidimensional Uncertainty-Aware Models

Robust Long-Term Hand Grasp Recognition With Raw Electromyographic Signals Using Multidimensional Uncertainty-Aware Models

i‐MYO: A multi‐grasp prosthetic hand control system based on gaze movements, augmented reality, and myoelectric signals

Grasp Pre-shape Selection by Synthetic Training: Eye-in-hand Shared Control on the Hannes Prosthesis

Contact Info

Product

Resources

About