Blending of brain-machine interface and vision-guided autonomous robotics improves neuroprosthetic arm performance during grasping

Abstract

BackgroundRecent studies have shown that brain-machine interfaces (BMIs) offer great potential for restoring upper limb function. However, grasping objects is a complicated task and the signals extracted from the brain may not always be capable of driving these movements reliably. Vision-guided robotic assistance is one possible way to improve BMI performance. We describe a method of shared control where the user controls a prosthetic arm using a BMI and receives assistance with positioning the hand when it approaches an object.MethodsTwo human subjects with tetraplegia used a robotic arm to complete object transport tasks with and without shared control. The shared control system was designed to provide a balance between BMI-derived intention and computer assistance. An autonomous robotic grasping system identified and tracked objects and defined stable grasp positions for these objects. The system identified when the user intended to interact with an object based on the BMI-controlled movements of the robotic arm. Using shared control, BMI controlled movements and autonomous grasping commands were blended to ensure secure grasps.ResultsBoth subjects were more successful on object transfer tasks when using shared control compared to BMI control alone. Movements made using shared control were more accurate, more efficient, and less difficult. One participant attempted a task with multiple objects and successfully lifted one of two closely spaced objects in 92 % of trials, demonstrating the potential for users to accurately execute their intention while using shared control.ConclusionsIntegration of BMI control with vision-guided robotic assistance led to improved performance on object transfer tasks. Providing assistance while maintaining generalizability will make BMI systems more attractive to potential users.Trial registrationNCT01364480 and NCT01894802.Electronic supplementary materialThe online version of this article (doi:10.1186/s12984-016-0134-9) contains supplementary material, which is available to authorized users.

Human neuroimaging reveals the subcomponents of grasping, reaching and pointing actions

“…Second, our findings provide additional clues with regard to the types of information available within human brain regions that could be exploited for the development of human neuromotor prosthetics that are sensitive to the wide variety of computations needed for dextrous hand actions (Aflalo et al, 2015;Andersen, Kellis, Klaes, & Aflalo, 2014;Collinger et al, 2013;Downey et al, 2016;Jarosiewicz et al, 2015).…”

From thought to action: The brain–machine interface in posterior parietal cortex

“…Neurons were also found that were extremely specialized for specific behavioral actions; for example, we found units that became active for imagined movements of the hand to the mouth but would not become active for similar movements such as movement of the hand to the cheek or forehead. Such neural encoding of behaviorally meaningful actions opens the possibility that the high-level intent of the subject can be decoded and integrated with smart robotics (49)(50)(51) to perform complex movements that may otherwise require attentionally demanding moment-to-moment control of a robotic limb. To demonstrate this concept for PPC, we showed that participant EGS was able to grasp an object and move it to a new location with a robotic limb, which combined his timing of the intended movements with machine vision and smart robotic algorithms (50).…”

Vision-Assisted Interactive Human-in-the-Loop Distal Upper Limb Rehabilitation Robot and its Clinical Usability Test

“…In this distal upper limb rehabilitation robot, the camera is mounted on the exoskeleton. This is different from other types of image-guided robot control devices, in which most of the cameras in other robots are placed externally in a fixed coordinate system [27,33]. This concept comes from a snake-eye view [34], in contrast to the human-eye view.…”

“…In this study, we attempted to apply an image processing-based approach that can reflect user-intent. Visual compensation using camera images in the BMI control of the robotic arm has recently been introduced [27]. Bang et al [28] suggested an upper limb rehabilitation robot system for precision control by camera-based image processing.…”