Hybrid Control of a Vision-Guided Robot Arm by EOG, EMG, EEG Biosignals and Head Movement Acquired via a Consumer-Grade Wearable Device

Abstract

Simultaneous acquisition of electrooculogram, jaw electromyogram, electroencephalogram, and head movement via consumer-grade wearable devices has become possible. Such devices offer new opportunities to deploy practical biosignal-based interfaces for assistive robots; however, they also pose challenges related to the available signals and their characteristics. In this proof-of-concept study, we demonstrate the possibility of successful control of a 5 + 1 degrees-of-freedom robot arm based on a consumer wireless headband in the form of four control modes predicated on distinct signal combinations. We propose a control approach hybrid at two levels, which seeks a compromise between robot controllability and maintaining the user goal rather than being process-focused. First, robot arm steering combines discrete and proportional aspects. Second, after the robot has been steered toward the approximate target direction, a sparse approach is followed and the user only needs to issue a single command, after which steering adjustment and grasping are performed automatically under stereoscopic vision guidance. We present in detail the associated algorithms, whose implementation is publicly available. Within this framework, we also demonstrate the control of arm posture and grasping force based, respectively, on object visual features and user input. We regard the interface proposed herein as a viable blueprint for future work on controlling wheelchair-mounted and mealassisting robot arms.INDEX TERMS Accelerometer, assistive robotics, brain-computer interface (BCI), brain-machine interface (BMI), electrooculogram (EOG), electroencephalogram (EEG), electromyogram (EMG), event-related desynchronization/synchronization (ERD/ERS), grasping force, human-machine interface (HMI), mealassisting robot, stereoscopic vision, wheelchair-mounted robot arm.

Feasibility and safety of shared EEG/EOG and vision-guided autonomous whole-arm exoskeleton control to perform activities of daily living

“…Performance of EEG/EOG control during use of this novel shared human-robot control strategy was comparable to other EEG/EOG control paradigms, e.g. asynchronous operation of a brain/neural hand exoskeleton 23 or a robotic arm 26 . None of the participants complained about any discomfort or undesirable effects.…”

Motor Imagery Based Continuous Teleoperation Robot Control with Tactile Feedback

“…The feasibility of inducing neurological recovery in paraplegic patients by long term training with a BCI-based gait protocol was shown in [5]. In addition, BCI-based control of virtual object [6], robotic arm [7][8][9], robotic prosthetic [10,11], wheelchair [12], and various rehabilitation devices [13][14][15][16] were also reported in previous research.…”

Multi-View Camera Pose Estimation for Robotic Arm Manipulation

“…Hence, each pose has to be accurate and in agreement to the overall geometric relationship for the 3D points to back-project precisely onto these different views from corresponding robot poses. The transformation is the same as used in (6) and shown in (4). The reprojection error is given as…”

“…Visual servoing uses visual information acquired from cameras to get spatial and semantic understanding of the surrounding to plan the motion of the robot. The most common applications are robotic grasping [6] and medical procedures [7]. Visual servoing depends on many independent components such as accuracy of robot positioning, hand-eye calibration, and target pose estimation.…”