“Wink to grasp” — comparing eye, voice & EMG gesture control of grasp with soft-robotic gloves

Abstract

The ability of robotic rehabilitation devices to support paralysed end-users is ultimately limited by the degree to which human-machine-interaction is designed to be effective and efficient in translating user intention into robotic action. Specifically, we evaluate the novel possibility of binocular eye-tracking technology to detect voluntary winks from involuntary blink commands, to establish winks as a novel low-latency control signal to trigger robotic action. By wearing binocular eye-tracking glasses we enable users to directly observe their environment or the actuator and trigger movement actions, without having to interact with a visual display unit or user interface. We compare our novel approach to two conventional approaches for controlling robotic devices based on electromyo-graphy (EMG) and speech-based human-computer interaction technology. We present an integrated software framework based on ROS that allows transparent integration of these multiple modalities with a robotic system. We use a soft-robotic SEM glove (Bioservo Technologies AB, Sweden) to evaluate how the 3 modalities support the performance and subjective experience of the end-user when movement assisted. All 3 modalities are evaluated in streaming, closed-loop control operation for grasping physical objects. We find that wink control shows the lowest error rate mean with lowest standard deviation of (0.23 ± 0.07, mean ± SEM) followed by speech control (0.35 ± 0. 13) and EMG gesture control (using the Myo armband by Thalamic Labs), with the highest mean and standard deviation (0.46 ± 0.16). We conclude that with our novel own developed eye-tracking based approach to control assistive technologies is a well suited alternative to conventional approaches, especially when combined with 3D eye-tracking based robotic end-point control.

Playing the piano with a robotic third thumb: Assessing constraints of human augmentation

“…We previously showed that the end-point of visual attention (where one looks) can control the spatial end-point of a robotic actuator with centimetre-level precision (Tostado et al, 2016;Maimon-Mor et al, 2017;Shafti et al, 2019b). This direct control modality is more effective from a user perspective than voice or neuromuscular signals as a natural control interface (Noronha et al, 2017). We showed that such direct augmentation can be used to control a supernumerary robotic arm to draw or paint, freeing up natural two arms to do other activities such as eating and drinking at the same time (Dziemian et al, 2016).…”

Fast orient: lightweight computer vision for wrist control in assistive robotic grasping

“…In our case, we demonstrated that our FastOrient algorithm works out-out-of-the-box which does not require a complex training learning pipeline and is sufficient to allow a natural wrist and grasp interaction. With the technology that we have demonstrated, we should be able to use gaze-based 3D positioning of the arm for grasping, as demonstrated in [4]- [6], wink-based detection of grasping intention [7], and now automatic processing of the grasp orientation based on the findings of this paper, without the requirement for the user to fine-tune the orientation of the hand, resulting in a full, intuitive assistive system. We have demonstrated FastOrient across 5 different realistic surfaces and for 26 different objects pertaining to standard activities of daily living.…”

“…Multi-modal systems relying on unaffected abilities, e.g. the use of gaze-based robotic end-point control for reaching assistance [4]- [6], or the use wearable robotics controlled through eye winks and voice [7]. In all these cases, a suitable eventual grasp is only possible with the correct orientation of the hand; be it the human hand using an orthotic ( Figure 1-left), or a tele-operated robotic hand, performing the grasp (Figure 1-right).…”

Offering a helping hand

“…It is plausible that results from healthy subjects are not transferable to the stroke population. In the study of Noronha et al [87], muscle activity was outperformed by wink and voice when assessing the error rate of grasping in healthy subjects. In the present study, we did not observe a statistical significant difference in success rate of the three modalities, but individual false positive rates were higher in wink controlled grasping when compared to the other modalities.…”

“…Muscle activity -In order to control the glove using myoelectric signals, a commercial [87]. The procedure used to control the glove by winking has previously been described by…”