Brain-Computer Interfaces (BCIs) enable users to interact with computers without any dedicated movement, bringing new hands-free interaction paradigms. In this paper we study the combination of BCI and Augmented Reality (AR). We first tested the feasibility of using BCI in AR settings based on Optical See-Through Head-Mounted Displays (OST-HMDs). Experimental results showed that a BCI and an OST-HMD equipment (EEG headset and Hololens in our case) are well compatible and that small movements of the head can be tolerated when using the BCI. Second, we introduced a design space for command display strategies based on BCI in AR, when exploiting a famous brain pattern called Steady-State Visually Evoked Potential (SSVEP). Our design space relies on five dimensions concerning the visual layout of the BCI menu ; namely: orientation, frame-of-reference, anchorage, size and explicitness. We implemented various BCI-based display strategies and tested them within the context of mobile robot control in AR. Our findings were finally integrated within an operational prototype based on a real mobile robot that is controlled in AR using a BCI and a HoloLens headset. Taken together our results (4 user studies) and our methodology could pave the way to future interaction schemes in Augmented Reality exploiting 3D User Interfaces based on brain activity and BCIs.

When persons interact with the environment and experience or witness an error (e.g. an unexpected event), a specific brain pattern, known as error-related potential (ErrP) can be observed in the electroencephalographic signals (EEG). Virtual Reality (VR) technology enables users to interact with computer-generated simulated environments and to provide multi-modal sensory feedback. Using VR systems can, however, be error-prone. In this paper, we investigate the presence of ErrPs when Virtual Reality users face 3 types of visualization errors: (Te) tracking errors when manipulating virtual objects, (Fe) feedback errors, and (Be) background anomalies. We conducted an experiment in which 15 participants were exposed to the 3 types of errors while performing a center-out pick and place task in virtual reality. The results showed that tracking errors generate error-related potentials, the other types of errors did not generate such discernible patterns. In addition, we show that it is possible to detect the ErrPs generated by tracking losses in single trial, with an accuracy of 85%. This constitutes a first step towards the automatic detection of error-related potentials in VR applications, paving the way to the design of adaptive and self-corrective VR/AR applications by exploiting information directly from the user's brain.

We present "Alfred", a novel haptic paradigm for object manipulation in 3D immersive virtual reality (VR). It uses a robotic manipulator to move tangible objects in its workspace such that they match the pose of virtual objects to be interacted with. Users can then naturally touch, grasp and manipulate a virtual object while feeling congruent and realistic haptic feedback from the tangible proxy. The tangible proxies can detach from the robot, allowing natural and unconstrained manipulation in the 3D virtual environment (VE). When a manipulated virtual object comes into contact with the virtual environment, the robotic manipulator acts as an encounter-type haptic display (ETHD), positioning itself so as to render reaction forces of the environment onto the manipulated physical object. Here, we discuss the concept for this novel approach and present a simplified prototype using a single detachable tangible proxy supported by a UR5 industrial robot. Through illustrative use-cases in VR and a preliminary performance evaluation, we discuss implications for robot control and design of interaction techniques. We show that Alfred is adaptable to a wide range of virtual environments and interaction scenarios, making it a promising approach for haptic-enabled manipulation in VR, although system latency is a limitation that still remains to be addressed.