Multimodal motor imagery decoding method based on temporal spatial feature alignment and fusion

Abstract

Objective: A motor imagery-based brain-computer interface (MI-BCI) translates spontaneous movement intention from the brain to outside devices. MI-BCI systems based on a single modality have been widely researched in recent decades. Lately, along with the development of neuroimaging methods, multimodal MI-BCI studies that use multiple neural signals have been proposed, which are promising for enhancing the decoding accuracy of MI-BCI. Multimodal MI data contain rich common and complementary information. Effective feature representations are helpful to promote the performance of classification tasks. Thus, it is very important to explore and extract features with higher separability and robustness from the rich information in multimodal data.
Approach: In this study, a five-class motor imagery experiment was designed. Electroencephalography and functional near infrared spectroscopy data were collected simultaneously. A multimodal MI decoding neural network was proposed. In this network, to enhance the feature representation, the heterogeneous data of different modalities in the spatial dimension were aligned through the proposed spatial alignment losses. Also, the multimodal features were aligned and fused in the temporal dimension by an attention-based modality fusion module.
Main results and Significance: The collected dataset was analyzed from temporal, spatial and frequency perspectives. The results showed that the multimodal data contain visually separable motor imagery patterns. The experimental results show that the proposed decoding method achieved the highest decoding accuracy among the compared methods on the self-collected dataset and a public dataset. Ablation results show that each part of the proposed method is effective. Compared with single-modality decoding, the proposed method obtained 4.6% higher decoding accuracy on the self-collected dataset. This indicates that the proposed method can improve the performance of multimodal MI decoding. This study provides a new approach for capturing the rich information in multimodal MI data and enhancing multimodal MI-BCI decoding accuracy.

A Study of Unilateral Upper Limb Fine Motor Imagery Decoding Using Frequency-Band Attention Network

“…Roy et al [17] proposed an efficient multiscale CNN (MS-CNN) with a classification accuracy of 93.74% on the BCI Competition IV-2a dataset. Zhang et al [18] designed a five-class motor imagery task involving imagining actions for the left hand, right hand, both hands, both feet, and rest. They concurrently collected EEG and fNIRS data.…”