Human-robot collaborative task planning using anticipatory brain responses

Ehrlich, Stefan K.; Dean-León, Emmanuel; Tacca, Nicholas; Armleder, Simon; Dimova-Edeleva, Viktorija; Cheng, Gordon

doi:10.1371/journal.pone.0287958

Cited by 4 publications

(1 citation statement)

References 86 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, in these studies, human users were merely passive observers of the error unlike in realworld applications. Recently, Ehrlich et al [28] showcased how EEG measures capture partner anticipation during task transitions, using these signals to train a reinforcement learning algorithm for dynamic subtask assignment in robot-human collaborations.…”

Section: Introductionmentioning

confidence: 99%

Prediction of cognitive conflict during unexpected robot behavior under different mental workload conditions in a physical human–robot collaboration

John,

Singh,

Gramann

et al. 2024

J. Neural Eng.

View full text Add to dashboard Cite

Brain-Computer Interface (BCI) technology is poised to play a prominent role in modern work environments, especially a collaborative environment where humans and machines work in close proximity, often with physical contact. In a physical human robot collaboration (pHRC), the robot performs complex motion sequences. Any unexpected robot behavior or faulty interaction might raise safety concerns. Error-related potentials, naturally generated by the brain when a human partner perceives an error, have been extensively employed in BCI as implicit human feedback to adapt robot behavior to facilitate a safe and intuitive interaction. As a higher workload on the user compromises their access to cognitive resources needed for error awareness, it is crucial to study how mental workload variations impact error awareness as it might raise safety concerns in pHRC. We designed a blasting task with an abrasive industrial robot and manipulated the mental workload with a secondary arithmetic task of varying difficulty. Electroencephalography (EEG) data, perceived workload, task and physical performance were recorded from twenty-four participants moving the robot arm. The error condition was achieved by the unexpected stopping of the robot in 33% of trials. We observed a diminished amplitude for the prediction error negativity (PEN) and error positivity (Pe), indicating reduced error awareness with increasing mental workload. We demonstrate that a popular convolution neural network model, EEGNet, could predict the amplitudes of PEN and Pe from the EEG data prior to the error. This prediction model could forewarn the system and operators of the diminished error awareness of the user, alluding to a potential safety breach in error-related potential-based BCI system for pHRC. Therefore, our work paves the way for embracing BCI technology in pHRC to optimally adapt the robot behavior for personalized user experience using real-time brain activity, enriching the quality of the interaction.

show abstract

Section: Introductionmentioning

confidence: 99%

Prediction of cognitive conflict during unexpected robot behavior under different mental workload conditions in a physical human–robot collaboration

John,

Singh,

Gramann

et al. 2024

J. Neural Eng.

View full text Add to dashboard Cite

show abstract

Wearable high-density EMG sleeve for complex hand gesture classification and continuous joint angle estimation

Tacca,

Dunlap,

Donegan

et al. 2024

Sci Rep

View full text Add to dashboard Cite

High-density electromyography (HD-EMG) can provide a natural interface to enhance human–computer interaction (HCI). This study aims to demonstrate the capability of a novel HD-EMG forearm sleeve equipped with up to 150 electrodes to capture high-resolution muscle activity, decode complex hand gestures, and estimate continuous hand position via joint angle predictions. Ten able-bodied participants performed 37 hand movements and grasps while EMG was recorded using the HD-EMG sleeve. Simultaneously, an 18-sensor motion capture glove calculated 23 joint angles from the hand and fingers across all movements for training regression models. For classifying across the 37 gestures, our decoding algorithm was able to differentiate between sequential movements with $$97.3 \pm 0.3\%$$ 97.3 ± 0.3 % accuracy calculated on a 100 ms bin-by-bin basis. In a separate mixed dataset consisting of 19 movements randomly interspersed, decoding performance achieved an average bin-wise accuracy of $$92.8 \pm 0.8\%$$ 92.8 ± 0.8 % . When evaluating decoders for use in real-time scenarios, we found that decoders can reliably decode both movements and movement transitions, achieving an average accuracy of $$93.3 \pm 0.9\%$$ 93.3 ± 0.9 % on the sequential set and $$88.5 \pm 0.9\%$$ 88.5 ± 0.9 % on the mixed set. Furthermore, we estimated continuous joint angles from the EMG sleeve data, achieving a $$R^2$$ R 2 of $$0.884 \pm 0.003$$ 0.884 ± 0.003 in the sequential set and $$0.750 \pm 0.008$$ 0.750 ± 0.008 in the mixed set. Median absolute error (MAE) was kept below 10° across all joints, with a grand average MAE of $$1.8 \pm 0.04^\circ$$ 1.8 ± 0 . 04 ∘ and $$3.4 \pm 0.07^\circ$$ 3.4 ± 0 . 07 ∘ for the sequential and mixed datasets, respectively. We also assessed two algorithm modifications to address specific challenges for EMG-driven HCI applications. To minimize decoder latency, we used a method that accounts for reaction time by dynamically shifting cue labels in time. To reduce training requirements, we show that pretraining models with historical data provided an increase in decoding performance compared with models that were not pretrained when reducing the in-session training data to only one attempt of each movement. The HD-EMG sleeve, combined with sophisticated machine learning algorithms, can be a powerful tool for hand gesture recognition and joint angle estimation. This technology holds significant promise for applications in HCI, such as prosthetics, assistive technology, rehabilitation, and human–robot collaboration.

show abstract

Challenges in Detecting and Analyzing EEG Error-Related Potentials: Lessons from a Case Study in HRI

Fava,

Lucchese,

Meattini

et al. 2024

2024 33rd IEEE International Conference on Robot and Human Interactive Communication (ROMAN)

View full text Add to dashboard Cite

Human-robot collaborative task planning using anticipatory brain responses

Cited by 4 publications

References 86 publications

Prediction of cognitive conflict during unexpected robot behavior under different mental workload conditions in a physical human–robot collaboration

Prediction of cognitive conflict during unexpected robot behavior under different mental workload conditions in a physical human–robot collaboration

Wearable high-density EMG sleeve for complex hand gesture classification and continuous joint angle estimation

Challenges in Detecting and Analyzing EEG Error-Related Potentials: Lessons from a Case Study in HRI

Contact Info

Product

Resources

About