Electrode-free visual prosthesis/exoskeleton control using augmented reality glasses in a first proof-of-technical-concept study

Abstract

In the field of neuroprosthetics, the current state-of-the-art method involves controlling the prosthesis with electromyography (EMG) or electrooculography/electroencephalography (EOG/EEG). However, these systems are both expensive and time consuming to calibrate, susceptible to interference, and require a lengthy learning phase by the patient. Therefore, it is an open challenge to design more robust systems that are suitable for everyday use and meet the needs of patients. In this paper, we present a new concept of complete visual control for a prosthesis, an exoskeleton or another end effector using augmented reality (AR) glasses presented for the first time in a proof-of-concept study. By using AR glasses equipped with a monocular camera, a marker attached to the prosthesis is tracked. Minimal relative movements of the head with respect to the prosthesis are registered by tracking and used for control. Two possible control mechanisms including visual feedback are presented and implemented for both a motorized hand orthosis and a motorized hand prosthesis. Since the grasping process is mainly controlled by vision, the proposed approach appears to be natural and intuitive.

3D Multi-Material Printing of an Anthropomorphic, Personalized Replacement Hand for Use in Neuroprosthetics Using 3D Scanning and Computer-Aided Design: First Proof-of-Technical-Concept Study

“…This requires that the user of the prosthesis have a healthy hand with which he can operate the two buttons on the remote control. A novel approach to electrodeless visual control of the orthosis with augmented reality glasses can counteract this [11,12], which is worth testing.…”

Metaverse Wearables for Immersive Digital Healthcare: A Review

“…[ 91 , 92 , 93 ] In addition, exoskeletons and prostheses linked with the XR platforms can increase rehabilitation efficiency and enable remote training. [ 94 , 95 , 96 ] A customized exoskeleton with cost‐effective fabrication and low power consumption has been developed to detect and display the joint movements of users ( Figure 6 a ). [ 97 ] The triboelectric bidirectional sensors embedded in the exoskeleton monitor multidimensional motions such as rotation, twisting, and linear motion.…”

A systematic review of technological advancements in signal sensing, actuation, control and training methods in robotic exoskeletons for rehabilitation

“…The results indicated a decrease in recovery time for patients undergoing training with AR technology. Hazubski et al (2020) demonstrated the application of AR glasses for tracking hand exoskeleton movements. The AR glass is used along with a monocular camera to track the movements.…”