EEG-fNIRS-based hybrid image construction and classification using CNN-LSTM

Abstract

The constantly evolving human–machine interaction and advancement in sociotechnical systems have made it essential to analyze vital human factors such as mental workload, vigilance, fatigue, and stress by monitoring brain states for optimum performance and human safety. Similarly, brain signals have become paramount for rehabilitation and assistive purposes in fields such as brain–computer interface (BCI) and closed-loop neuromodulation for neurological disorders and motor disabilities. The complexity, non-stationary nature, and low signal-to-noise ratio of brain signals pose significant challenges for researchers to design robust and reliable BCI systems to accurately detect meaningful changes in brain states outside the laboratory environment. Different neuroimaging modalities are used in hybrid settings to enhance accuracy, increase control commands, and decrease the time required for brain activity detection. Functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) measure the hemodynamic and electrical activity of the brain with a good spatial and temporal resolution, respectively. However, in hybrid settings, where both modalities enhance the output performance of BCI, their data compatibility due to the huge discrepancy between their sampling rate and the number of channels remains a challenge for real-time BCI applications. Traditional methods, such as downsampling and channel selection, result in important information loss while making both modalities compatible. In this study, we present a novel recurrence plot (RP)-based time-distributed convolutional neural network and long short-term memory (CNN-LSTM) algorithm for the integrated classification of fNIRS EEG for hybrid BCI applications. The acquired brain signals are first projected into a non-linear dimension with RPs and fed into the CNN to extract essential features without performing any downsampling. Then, LSTM is used to learn the chronological features and time-dependence relation to detect brain activity. The average accuracies achieved with the proposed model were 78.44% for fNIRS, 86.24% for EEG, and 88.41% for hybrid EEG-fNIRS BCI. Moreover, the maximum accuracies achieved were 85.9, 88.1, and 92.4%, respectively. The results confirm the viability of the RP-based deep-learning algorithm for successful BCI systems.

Deep learning-based framework for real-time upper limb motion intention classification using combined bio-signals

“…The present study validates the advantages of a hybrid neural-machine interface (Mughal et al, 2022 ). The hybridization of two distinct modalities resulted in stable output performance, unlike the results reported in published studies for a single modality source.…”

Graph neural network based on brain inspired forward-forward mechanism for motor imagery classification in brain-computer interfaces

“…In recent times, DL has achieved remarkable performance across various fields due to its ability to extract underlying features from signals, thus mitigating the necessity for manual feature engineering. The Convolutional Neural Network (CNN) has found extensive usage in classifying Euclidean-structured signals, credited to its capability to learn informative features through local receptive fields (Du Y. et al, 2022;Mughal et al, 2022;de Oliveira and Rodrigues, 2023). Tang et al (2023a) developed a Multi-Scale Hybrid CNN to isolate temporal and spatial EEG signal attributes.…”

Novel Feature Generation for Classification of Motor Activity from Functional Near-Infrared Spectroscopy Signals Using Machine Learning

“…Huve et al [28] implemented classification based on support vector machine (SVM), long short-term memory (LSTM), and LDA for tasks such as counting, arithmetic, and solving puzzles. Classification of the fNIRS signal based on three mental tasks such as rest state, subtraction, and generation of words using LSTM and CNN was studied by Asgher [29,30]; their results were improved and compared to classical machine learning methods [31]. Recurrent CNN showed classification of both time and spatial signals with good accuracy of EEG signals [32].…”