Audio-visual feedback improves the BCI performance in the navigational control of a humanoid robot

Abstract

Advancement in brain computer interfaces (BCI) technology allows people to actively interact in the world through surrogates. Controlling real humanoid robots using BCI as intuitively as we control our body represents a challenge for current research in robotics and neuroscience. In order to successfully interact with the environment the brain integrates multiple sensory cues to form a coherent representation of the world. Cognitive neuroscience studies demonstrate that multisensory integration may imply a gain with respect to a single modality and ultimately improve the overall sensorimotor performance. For example, reactivity to simultaneous visual and auditory stimuli may be higher than to the sum of the same stimuli delivered in isolation or in temporal sequence. Yet, knowledge about whether audio-visual integration may improve the control of a surrogate is meager. To explore this issue, we provided human footstep sounds as audio feedback to BCI users while controlling a humanoid robot. Participants were asked to steer their robot surrogate and perform a pick-and-place task through BCI-SSVEPs. We found that audio-visual synchrony between footsteps sound and actual humanoid's walk reduces the time required for steering the robot. Thus, auditory feedback congruent with the humanoid actions may improve motor decisions of the BCI's user and help in the feeling of control over it. Our results shed light on the possibility to increase robot's control through the combination of multisensory feedback to a BCI user.

Hybrid Brain–Computer Interface Techniques for Improved Classification Accuracy and Increased Number of Commands: A Review

“…Although mostly such stimulations are intended for the generation of brain activity from the corresponding lobes (An et al, 2014; Tidoni et al, 2014; Wu et al, 2016; Xu et al, 2016), some audio/video stimuli are given to generate P300 signals (Rutkowski, 2016). For healthy individuals, these stimulations can be effective in generating multiple commands.…”

Agency and responsibility over virtual movements controlled through different paradigms of brain–computer interface

“…Remotely BCI-controlled androids are a promising possibility for the world and for the social interaction of immobilized patients, and the sense of control and responsibility over, for example, limb prostheses and more remote effectors (Tidoni, Gergondet, Kheddar, & Aglioti, 2014), should be a central consideration when developing these systems. This is a topic of high ethical relevance (Yuste et al, 2017), since we should expect in the next few decades a considerable growth of different forms of BCIs that can control anything from a screen cursor to a robot in another continent (Tidoni et al, 2014). A thorough understanding of the responsibility felt for these actions and its neural basis is therefore of the utmost importance.…”

Audible Feedback Improves Internal Model Strength and Performance of Myoelectric Prosthesis Control

“…Conveniently, augmented feedback can be used to convey artificial proprioceptive and exteroceptive information 21,29 , which may help to develop strong internal models. Researchers have used audio augmented feedback in both robotic teleoperation 30,31 and Brain Computer Interfaces (BCI) 32 and have concluded that audio augmented feedback improves performance. Unlike visual feedback, audio requires less focus of attention and reduces distraction 33,34 .…”