Navigating a 2D Virtual World Using Direct Brain Stimulation

Abstract

Can the human brain learn to interpret inputs from a virtual world delivered directly through brain stimulation? We answer this question by describing the first demonstration of humans playing a computer game utilizing only direct brain stimulation and no other sensory inputs. The demonstration also provides the first instance of artificial sensory information, in this case depth, being delivered directly to the human brain through non-invasive methods. Our approach utilizes transcranial magnetic stimulation (TMS) of the human visual cortex to convey binary information about obstacles in a virtual maze. At certain intensities, TMS elicits visual percepts known as phosphenes, which transmit information to the subject about their current location within the maze. Using this computer-brain interface, five subjects successfully navigated an average of 92% of all the steps in a variety of virtual maze worlds. They also became more accurate in solving the task over time. These results suggest that humans can learn to utilize information delivered directly and non-invasively to their brains to solve tasks that cannot be solved using their natural senses, opening the door to human sensory augmentation and novel modes of human-computer interaction.

BrainNet: A Multi-Person Brain-to-Brain Interface for Direct Collaboration Between Brains

“…As noted in previous studies from our group 1,5,14 , raw accuracy can be an inadequate measure of performance because it does not differentiate the kind of mistakes being made, i.e., whether they are misses or false positives. A better measure of performance can be obtained by calculating each triad's Receiver Operating Characteristic (ROC) curve 15 , which plots the True Positive Rate versus the False Positive rate, and calculating the area under this curve (AUC; Figure 5).…”

Repeated Application of Transcranial Diagnostic Ultrasound Towards the Visual Cortex Induced Illusory Visual Percepts in Healthy Participants

“…Nonetheless, we think this unlikely in our case for a number of reasons. First, our thresholding procedures adhered to the published guidelines that have consistently found no carry-over effects between single pulses separated by at least 5 s. In fact, the very existence of thresholding procedures such as the one employed in this study and previously used in other studies (Stocco et al, 2015;Losey et al, 2016;Jiang et al, 2019) depends on the documented lack of carry-over effects. Second, all of the established TMS protocols designed to have long-term effects (Ridding and Ziemann, 2010;Stagg and Nitsche, 2011) require many more pulses per second (e.g., 1 Hz or higher) and/or much longer exposure (e.g., 300 or 600 pulses).…”

“…Motivated by observations of alterations of brain function discernible by the participants themselves (Hameroff et al, 2013;Legon et al, 2014;Mueller et al, 2014;Lee et al, 2015Lee et al, , 2016aLosey et al, 2016;Gibson et al, 2018), including by DU, we sought as our primary goal to determine if DU could alter the visual perception of a target (that is, their ''percept'') when applied at the anatomical location where TMS could induce phosphenes. As a secondary goal, we sought to determine if the hypothesized effects on the visual cortex arose due to the short-term or long-term effects of ultrasound exposure.…”

Technology Integration Methods for Bi-directional Brain-computer Interfaces and XR-based Interventions

“…XR-based interventions also present potential utility for functional neurosurgery patients [ 19 ], providing a controlled environment and a reported increased level of understanding, comfort, and satisfaction among patients [ 20 ]. The untapped potential of combining virtual environments and brain stimulation has led to an influx of new research, creating a new technological niche [ 21 ]–[ 23 ]. Game-based interventions including exergames and XR applications have been integrated with electroencephalography (EEG) and transcranial alternating current stimulation (tACS) for recording and monitoring purposes [ 24 ]–[ 26 ].…”