A service assistant combining autonomous robotics, flexible goal formulation, and deep-learning-based brain–computer interfacing

Kuhner, Daniel; Fiederer, Lukas Dominique Josef; Aldinger, Johannes; Burget, Felix; Volker, Martin A.; Schirrmeister, Robin Tibor; Do, Chau; Boedecker, Joschka; Nebel, Bernhard; Ball, Tonio; Burgard, Wolfram

doi:10.1016/j.robot.2019.02.015

Cited by 41 publications

(27 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Nowadays, the advance of the Internet of Things (IoT) is making the quick use of pretty specific devices even more demanding of direct guiding [49]. The control of complex robotic tasks through touch interfaces, the use of brain-computer interfaces, especially for disabled people, does make guiding the user a necessity [50].…”

Section: Discussionmentioning

confidence: 99%

Proposal and testing goals-guided interaction for occasional users

Carrillo

Falgueras

2020

Hum. Cent. Comput. Inf. Sci.

View full text Add to dashboard Cite

One of the main research areas of human-computer interaction is the study of the different ways in which users communicate or interact with the computer [1, 2]. Each interaction style offers its own way of organising system functionality, managing user inputs, and displaying information. Two main approaches can be considered in order to interact with modern devices: the conversational world and the model world. The former is sequential and based on text. The latter, the model world, uses graphics and metaphors [3], like "Windows, Icons, Menus and Pointers" (WIMP), to assist the user with an asynchronous and a free management of objects on the screen. Users can see and predict the behaviour of familiar objects through metaphors. They then follow their natural intuition to manipulate them, receiving immediate feedback. The success of this Direct

show abstract

Section: Discussionmentioning

confidence: 99%

Proposal and testing goals-guided interaction for occasional users

Carrillo

Falgueras

2020

Hum. Cent. Comput. Inf. Sci.

View full text Add to dashboard Cite

show abstract

“…Including the training data of the tested user into a hybrid within/across-user training could also further improve regression performance. Although such data would not be directly available in an online BCI scenario an initial across-user model could be fine-tuned as more and more labeled data of a new user become available during an online experiment as e.g., we have done for online-adaptive classification using CNNs in Kuhner et al (2019).…”

Section: Discussionmentioning

confidence: 99%

“…After pioneering achievements in the field of computer vision, they are increasingly being adapted to problems of EEG decoding (Manor and Geva, 2015; Bashivan et al, 2016) and are the subject of active research (e.g., Eitel et al, 2015; Watter et al, 2015; Oliveira et al, 2016). These biologically inspired networks have a great potential to improve the accuracy of BCI applications (Burget et al, 2017; Schirrmeister et al, 2017; Kuhner et al, 2019). They additionally can be applied to the raw EEG data, greatly simplifying the design of BCI pipelines.…”

Section: Introductionmentioning

confidence: 99%

Hybrid Brain-Computer-Interfacing for Human-Compliant Robots: Inferring Continuous Subjective Ratings With Deep Regression

et al. 2019

View full text Add to dashboard Cite

Appropriate robot behavior during human-robot interaction is a key part in the development of human-compliant assistive robotic systems. This study poses the question of how to continuously evaluate the quality of robotic behavior in a hybrid brain-computer interfacing (BCI) task, combining brain and non-brain signals, and how to use the collected information to adapt the robot's behavior accordingly. To this aim, we developed a rating system compatible with EEG recordings, requiring the users to execute only small movements with their thumb on a wireless controller to rate the robot's behavior on a continuous scale. The ratings were recorded together with dry EEG, respiration, ECG, and robotic joint angles in ROS. Pilot experiments were conducted with three users that had different levels of previous experience with robots. The results demonstrate the feasibility to obtain continuous rating data that give insight into the subjective user perception during direct human-robot interaction. The rating data suggests differences in subjective perception for users with no, moderate, or substantial previous robot experience. Furthermore, a variety of regression techniques, including deep CNNs, allowed us to predict the subjective ratings. Performance was better when using the position of the robotic hand than when using EEG, ECG, or respiration. A consistent advantage of features expected to be related to a motor bias could not be found. Across-user predictions showed that the models most likely learned a combination of general and individual features across-users. A transfer of pre-trained regressor to a new user was especially accurate in users with more experience. For future research, studies with more participants will be needed to evaluate the methodology for its use in practice. Data and code to reproduce this study are available at https://github.com/TNTLFreiburg/NiceBot.

show abstract

“…The MI paradigm has advanced steadily, and the neurophysiological phenomena (e.g., event-related desynchronization and synchronization (ERD/ERS)) during motor execution (ME) are observed when MI is performed. ERD/ERS rhythms are identified from the mu-band (8)(9)(10)(11)(12) and the beta-band (13)(14)(15)(16)(17)(18)(19)(20)(21)(22)(23)(24)(25)(26)(27)(28)(29)(30) across the primary sensorimotor area [22]. However, the ERD/ERS pattern can detect differences in band-specific signal patterns and the location of the occurrence depending on the individual characteristics [23].…”

Section: Introductionmentioning

confidence: 99%

EEG Classification of Forearm Movement Imagery Using a Hierarchical Flow Convolutional Neural Network

Jeong

Lee

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Recent advances in brain-computer interface (BCI) techniques have led to increasingly refined interactions between users and external devices. Accurately decoding kinematic information from brain signals is one of the main challenges encountered in the control of human-like robots. In particular, although the forearm of an upper extremity is frequently used in daily life for high-level tasks, only few studies addressed decoding of the forearm movement. In this study, we focus on the classification of forearm movements according to elaborated rotation angles using electroencephalogram (EEG) signals. To this end, we propose a hierarchical flow convolutional neural network (HF-CNN) model for robust classification. We evaluate the proposed model not only with our experimental dataset but also with a public dataset (BNCI Horizon 2020). The grand-average classification accuracies of three rotation angles yield 0.73 (±0.04) for the motor execution (ME) task and 0.65 (±0.09) for the motor imagery (MI) task across ten subjects in our experimental dataset. Further, in the public dataset, the grand-averaged classification accuracies were 0.52 (±0.03) for ME and 0.51 (±0.04) for MI tasks across fifteen subjects. Our experimental results demonstrate the possibility of decoding complex kinematics information using EEG signals. This study will contribute to the development of a brain-controlled robotic arm system capable of performing high-level tasks. INDEX TERMS Brain-computer interface (BCI), electroencephalogram (EEG), convolutional neural network (CNN), forearm motor execution and motor imagery.

show abstract

A service assistant combining autonomous robotics, flexible goal formulation, and deep-learning-based brain–computer interfacing

Cited by 41 publications

References 34 publications

Proposal and testing goals-guided interaction for occasional users

Proposal and testing goals-guided interaction for occasional users

Hybrid Brain-Computer-Interfacing for Human-Compliant Robots: Inferring Continuous Subjective Ratings With Deep Regression

EEG Classification of Forearm Movement Imagery Using a Hierarchical Flow Convolutional Neural Network

Contact Info

Product

Resources

About