On the Visuomotor Behavior of Amputees and Able-Bodied People During Grasping

Abstract

Visual attention is often predictive for future actions in humans. In manipulation tasks, the eyes tend to fixate an object of interest even before the reach-to-grasp is initiated. Some recent studies have proposed to exploit this anticipatory gaze behavior to improve the control of dexterous upper limb prostheses. This requires a detailed understanding of visuomotor coordination to determine in which temporal window gaze may provide helpful information. In this paper, we verify and quantify the gaze and motor behavior of 14 transradial amputees who were asked to grasp and manipulate common household objects with their missing limb. For comparison, we also include data from 30 able-bodied subjects who executed the same protocol with their right arm. The dataset contains gaze, first person video, angular velocities of the head, and electromyography and accelerometry of the forearm. To analyze the large amount of video, we developed a procedure based on recent deep learning methods to automatically detect and segment all objects of interest. This allowed us to accurately determine the pixel distances between the gaze point, the target object, and the limb in each individual frame. Our analysis shows a clear coordination between the eyes and the limb in the reach-to-grasp phase, confirming that both intact and amputated subjects precede the grasp with their eyes by more than 500 ms. Furthermore, we note that the gaze behavior of amputees was remarkably similar to that of the able-bodied control group, despite their inability to physically manipulate the objects.

Gesture Recognition Using Surface Electromyography and Deep Learning for Prostheses Hand: State-of-the-Art, Challenges, and Future

“…The study of different algorithms for intact subjects can provide a lot of valuable information to help achieve better prosthetic performance (Merad et al, 2018 ). Even so, it is essential to utilize the data collected from amputees (Gregori et al, 2019 ). Intact subjects performed real movements of the hand, while amputees could only attempt imaginary movements without visual and sensory feedback, so that when they perform the same movements at a specific arm position, the residual muscles of amputated arm might produce sEMG patterns different from those of intact arm (Geng et al, 2012 ; Vidovic et al, 2016 ).…”

Improving Robotic Hand Prosthesis Control With Eye Tracking and Computer Vision: A Multimodal Approach Based on the Visuomotor Behavior of Grasping

“…It is based on the implementation provided by Massa and Girshick ( 2018 ) that was originally trained on the Common Objects in Context (COCO) dataset (Lin et al, 2014 ) and fine-tuned on the MeganePro objects. Gregori et al ( 2019 ) demonstrated the substantial increase in average precision of this model with respect to the non-fine-tuned model when tested on the on MeganePro objects, thanks also to the limited variability and number of objects employed in the MeganePro acquisitions. To reduce the computation time, we extracted and stored the contour of the objects identified by this network only from 2 s before to 3.5 s after the beginning of the grasp identified from the relabeled data.…”

“…Object recognition and segmentation were performed with a Mask R-CNN network (He et al, 2017 ), utilizing the model released by Gregori et al ( 2019 ). The model uses a ResNet-50-Feature Pyramid Network (He et al, 2016 ; Lin et al, 2017 ) as backbone.…”

“…To reduce the computation time, we extracted and stored the contour of the objects identified by this network only from 2 s before to 3.5 s after the beginning of the grasp identified from the relabeled data. Furthermore, as done in Gregori et al ( 2019 ), instances having a score level lower than 0.8 were discarded. At this stage, the gaze points within each video frame (at their original sampling rate) were used to identify the object being looked at by the subject.…”

“…Second, the approach was not fully based on deep neural networks. Third, it included the identification of fixations, which brings the disadvantage of shortening the time frame to identify the target object, that could prevent a correct identification (Gregori et al, 2019 ). Evaluations on transradial amputees are particularly desirable to better investigate the performance of multimodal approaches based on gaze and sEMG for prosthetic applications.…”