Magnetoencephalogram-based brain–computer interface for hand-gesture decoding using deep learning

Abstract

Advancements in deep learning algorithms over the past decade have led to extensive developments in brain–computer interfaces (BCI). A promising imaging modality for BCI is magnetoencephalography (MEG), which is a non-invasive functional imaging technique. The present study developed a MEG sensor-based BCI neural network to decode Rock-Paper-scissors gestures (MEG-RPSnet). Unique preprocessing pipelines in tandem with convolutional neural network deep-learning models accurately classified gestures. On a single-trial basis, we found an average of 85.56% classification accuracy in 12 subjects. Our MEG-RPSnet model outperformed two state-of-the-art neural network architectures for electroencephalogram-based BCI as well as a traditional machine learning method, and demonstrated equivalent and/or better performance than machine learning methods that have employed invasive, electrocorticography-based BCI using the same task. In addition, MEG-RPSnet classification performance using an intra-subject approach outperformed a model that used a cross-subject approach. Remarkably, we also found that when using only central-parietal-occipital regional sensors or occipitotemporal regional sensors, the deep learning model achieved classification performances that were similar to the whole-brain sensor model. The MEG-RSPnet model also distinguished neuronal features of individual hand gestures with very good accuracy. Altogether, these results show that noninvasive MEG-based BCI applications hold promise for future BCI developments in hand-gesture decoding.

EMG-projected MEG High-Resolution Source Imaging of Human Motor Execution: Brain-Muscle Coupling above Movement Frequencies

“…Negative gamma-band MR-CMC: Gamma-band EMG-projected MEG source imaging failed to localize M1 activity during right and left index finger movement in any of the 13 healthy subjects. This negative result is compatible with our MEG-based brain-computer interface study of decoding hand gestures, for which the gamma-band activity did not contribute to hand-gesture classification accuracy (Bu et al, 2023). In other studies, MEG-based gamma band M1 activity was mainly found post-movement-onset during repetitive finger tapping movements using a seed virtual sensor placed at M1 pre-located from evoked-related finger movement components (Cheyne et al, 2008;Huo et al, 2010).…”

“…Nonetheless, our negative findings should be interpreted with caution, since significant coherence between ECoG and EMG in low (31-60 Hz) and high gamma (61-100 Hz) bands originates from M1 cortex (Marsden et al, 2000). The absence of gamma-band EMG-M1 coupling in our study may be due to lower SNR in the non-invasive MEG gamma band than for invasive ECoG recordings (Bu et al, 2023).…”

“…Regardless of the modeling approach, however, few studies have systematically examined movement-related M1 cortex activity across different frequency bands, including lower bands, which influence M1 cortex activity (e.g., Popovych et al, 2016). Moreover, M1 cortex activity typically is not just limited to muscle activity, and non-motor brain activities from background and artifacts may affect brain-muscle communication (Bu et al, 2023).…”

Brain-computer interface paradigms and neural coding

“…Magnetoencephalography is a non-invasive neuroimaging technique for detecting weak magnetic field changes generated by the electrical activity of central neurons ( Xu et al, 2022 ; Bu et al, 2023 ). The technique has high time (less than 1 ms) and spatial [2–5 μm ( Cetin and Temurtas, 2021 )] resolution and low sensitivity to artifacts generated by muscle activity, but its comfort, aesthetics, and ease of use leave much to be desired.…”

“…For example, Abiri et al (2019) reviewed EEG-based BCI paradigms, and Xu et al (2021) reviewed the EEG-based BCI brain coding and decoding mechanisms. In addition to EEG-based BCI paradigms and neural coding, there are also other BCI paradigms and neural coding based on brain imaging techniques, such as intracortical local field potentials (LFP) ( Hochberg et al, 2012 ; Willett et al, 2021 , 2023 ), electroencephalogram (ECoG) ( Luo et al, 2022 ; Branco et al, 2023 ; Metzger et al, 2023 ), functional near-infrared spectroscopy (fNIRS) ( Abdalmalak et al, 2021 ; Paulmurugan et al, 2021 ; Eastmond et al, 2022 ), functional magnetic resonance imaging (fMRI) ( Naselaris et al, 2011 ; Du et al, 2019 ), magnetoencephalography (MEG) ( Xu et al, 2022 ; Bu et al, 2023 ), and hybrid brain-computer interface (hBCI) ( Choi et al, 2017 ; Mussi and Adams, 2022 ). Therefore, we systematically elaborated on the definition and design principles of the BCI paradigm as well as the definition and modeling principles of BCI neural coding and introduced the existing main BCI paradigms and neural coding.…”