Brain–Computer Interface (BCI) Control of a Virtual Assistant in a Smartphone to Manage Messaging Applications

Abstract

Brain–computer interfaces (BCI) are a type of assistive technology that uses the brain signals of users to establish a communication and control channel between them and an external device. BCI systems may be a suitable tool to restore communication skills in severely motor-disabled patients, as BCI do not rely on muscular control. The loss of communication is one of the most negative consequences reported by such patients. This paper presents a BCI system focused on the control of four mainstream messaging applications running in a smartphone: WhatsApp, Telegram, e-mail and short message service (SMS). The control of the BCI is achieved through the well-known visual P300 row-column paradigm (RCP), allowing the user to select control commands as well as spelling characters. For the control of the smartphone, the system sends synthesized voice commands that are interpreted by a virtual assistant running in the smartphone. Four tasks related to the four mentioned messaging services were tested with 15 healthy volunteers, most of whom were able to accomplish the tasks, which included sending free text e-mails to an address proposed by the subjects themselves. The online performance results obtained, as well as the results of subjective questionnaires, support the viability of the proposed system.

Brain-computer interface (BCI)-generated speech to control domotic devices

“…A comparison with our previous study that followed a similar procedure (i.e., the use of voice commands built using a BCI system for interpretation by a voice assistant to control diverse messaging systems [27]) shows that the accuracy obtained here was slightly lower (80.68 % vs 86.14 %), but the information transfer rate (ITR) was higher (25.9 bits/min vs 21.69bits/min). The ITR is the number of bits transmitted per second and provides a more general evaluation than accuracy since, besides considering it, it takes into account the number of elements available in the interface as well as the time needed for each selection, which depends on the number of sequences used [42].…”

“…Based on the accuracy of the results obtained with this classifier, the number of sequences was adjusted for each subject. For healthy participants, the criterion applied to choose the number of sequences aimed to maximize the written symbol rate (WSR), as in [27,34,35]. In the same way as in [27], this criterion was adapted so that the minimum number of sequences was three.…”

“…For healthy participants, the criterion applied to choose the number of sequences aimed to maximize the written symbol rate (WSR), as in [27,34,35]. In the same way as in [27], this criterion was adapted so that the minimum number of sequences was three. However, for the patients, the choice of the number of sequences was made subjectively, based on the one with which the user felt comfortable, as long as the accuracy was 100 % in the calibration step.…”

“…To our knowledge, this idea of bridging communication between the BCI system and other devices (or applications) through voice commands has previously been employed in only two works. In [27], a BCI messaging system was created to control Whatsapp, Telegram, an email client and an SMS system via UMA-BCI Speller and Google Assistant. UMA-BCI Speller is easily configurable software that uses a visual ERP signal and allows for flexibility in terms of creating the graphical user interface, navigating between menus and freely constructing the desired voice commands [28].…”

Denoising Autoencoder-Based Feature Extraction to Robust SSVEP-Based BCIs

“…BCIs had been widely developed to allow subjects to control devices or communicate with others by modulating their brain signals [8][9][10][11][12][13][14][15][16][17][18]. Generally, the BCIs use near-infrared spectroscopy, functional magnetic resonance imaging, magnetoencephalography, or electroencephalogram (EEG) to monitor a user's brain activities [8][9][10]. Using the EEG as the recording methods has a relatively lower cost [11,12], thus the EEG had been successfully used in BCIs.…”