Multimodal Autoencoder Predicts fNIRS Resting State From EEG Signals

Sirpal, Parikshat; Damseh, Rafat; Peng, Ke; Nguyen, Dang Khoa; Lesage, Frédéric

doi:10.1007/s12021-021-09538-3

Cited by 22 publications

(6 citation statements)

References 126 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Combining fNIRS with EEG is beneficial, since EEG can measure neuronal activity at a high temporal resolution for microstate analysis. In contrast, fNIRS can uncover cortical correlates of microstates under the neurovascular coupling phenomenon [49,88,90]. Microstate prototypes were selected from the excellent temporal resolution of EEG [64] and the meta-criterion for global field power (GFP) [16].…”

Section: Introductionmentioning

confidence: 99%

Error-related brain state analysis using electroencephalography in conjunction with functional near-infrared spectroscopy during a complex surgical motor task

Walia

Fu²,

Norfleet³

et al. 2022

Brain Inf.

View full text Add to dashboard Cite

Error-based learning is one of the basic skill acquisition mechanisms that can be modeled as a perception–action system and investigated based on brain–behavior analysis during skill training. Here, the error-related chain of mental processes is postulated to depend on the skill level leading to a difference in the contextual switching of the brain states on error commission. Therefore, the objective of this paper was to compare error-related brain states, measured with multi-modal portable brain imaging, between experts and novices during the Fundamentals of Laparoscopic Surgery (FLS) “suturing and intracorporeal knot-tying” task (FLS complex task)—the most difficult among the five psychomotor FLS tasks. The multi-modal portable brain imaging combined functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) for brain–behavior analysis in thirteen right-handed novice medical students and nine expert surgeons. The brain state changes were defined by quasi-stable EEG scalp topography (called microstates) changes using 32-channel EEG data acquired at 250 Hz. Six microstate prototypes were identified from the combined EEG data from experts and novices during the FLS complex task that explained 77.14% of the global variance. Analysis of variance (ANOVA) found that the proportion of the total time spent in different microstates during the 10-s error epoch was significantly affected by the skill level (p < 0.01), the microstate type (p < 0.01), and the interaction between the skill level and the microstate type (p < 0.01). Brain activation based on the slower oxyhemoglobin (HbO) changes corresponding to the EEG band power (1–40 Hz) changes were found using the regularized temporally embedded Canonical Correlation Analysis of the simultaneously acquired fNIRS–EEG signals. The HbO signal from the overlying the left inferior frontal gyrus—opercular part, left superior frontal gyrus—medial orbital, left postcentral gyrus, left superior temporal gyrus, right superior frontal gyrus—medial orbital cortical areas showed significant (p < 0.05) difference between experts and novices in the 10-s error epoch. We conclude that the difference in the error-related chain of mental processes was the activation of cognitive top-down attention-related brain areas, including left dorsolateral prefrontal/frontal eye field and left frontopolar brain regions, along with a ‘focusing’ effect of global suppression of hemodynamic activation in the experts, while the novices had a widespread stimulus(error)-driven hemodynamic activation without the ‘focusing’ effect.

show abstract

Section: Introductionmentioning

confidence: 99%

Error-related brain state analysis using electroencephalography in conjunction with functional near-infrared spectroscopy during a complex surgical motor task

Walia

Fu²,

Norfleet³

et al. 2022

Brain Inf.

View full text Add to dashboard Cite

show abstract

“…7 , with the CNN achieving an area under the curve (AUC) of the receiver operating characteristic curve (ROC curve) of 0.92, which outperformed the ANN, ICA, and seed-based methods with each reporting an AUC of 0.89, 0.88, and 0.79, respectively. Another study that investigated RSFC, Sirpal et al., 85 collected EEG and fNIRS data and attempted to use the EEG data and an LSTM to recreate the fNIRS signals. Using only the gamma bands of the EEG signal was found to have the lowest reconstruction error, below 0.25.…”

Section: Applications In Fnirsmentioning

confidence: 99%

Deep learning in fNIRS: a review

Eastmond

Subedi

et al. 2022

Neurophoton.

View full text Add to dashboard Cite

Significance: Optical neuroimaging has become a well-established clinical and research tool to monitor cortical activations in the human brain. It is notable that outcomes of functional nearinfrared spectroscopy (fNIRS) studies depend heavily on the data processing pipeline and classification model employed. Recently, deep learning (DL) methodologies have demonstrated fast and accurate performances in data processing and classification tasks across many biomedical fields.Aim: We aim to review the emerging DL applications in fNIRS studies.Approach: We first introduce some of the commonly used DL techniques. Then, the review summarizes current DL work in some of the most active areas of this field, including braincomputer interface, neuro-impairment diagnosis, and neuroscience discovery.Results: Of the 63 papers considered in this review, 32 report a comparative study of DL techniques to traditional machine learning techniques where 26 have been shown outperforming the latter in terms of the classification accuracy. In addition, eight studies also utilize DL to reduce the amount of preprocessing typically done with fNIRS data or increase the amount of data via data augmentation. Conclusions:The application of DL techniques to fNIRS studies has shown to mitigate many of the hurdles present in fNIRS studies such as lengthy data preprocessing or small sample sizes while achieving comparable or improved classification accuracy.

show abstract

“…This helps an MAE [45,47,48] to learn from the prior distribution flexibly by capturing features from a target distribution. Therefore, MAE-based approaches have been applied in a variety of settings such as natural language understanding (e.g., document and dialogue modeling) [45], emotion recognition [47], and near-infrared spectroscopy resting state prediction from multimodal electroencephalographic signals [49]. Besides, to make the most of multimodal data, a large amount of literature is devoted to the construction of integration methods for predicting cancer survival [48].…”

Section: B Multimodal Approachesmentioning

confidence: 99%

Adversary-Aware Multimodal Neural Networks for Cancer Susceptibility Prediction From Multiomics Data

et al. 2022

View full text Add to dashboard Cite

Artificial intelligence (AI) systems are increasingly used in health and personalized care. However, the adoption of data-driven approaches in many clinical settings has been hampered due to their inability to perform in a reliable and safe manner to leverage accurate and trustworthy diagnoses. A critical and challenging usage scenario for AI is aiding the treatment of cancerous conditions. Providing accurate diagnosis for cancer is a challenging problem in precision oncology. Although machine learning (ML)-based approaches are very effective at cancer susceptibility prediction and subsequent treatment recommendations, ML models can be vulnerable to adversarial attacks. Since adversarially weak models can lead to wrong clinical recommendations, such vulnerabilities is more critical -especially when AI-guided systems are used to aid medical doctors. Therefore, it is indispensable that healthcare professionals employ trustworthy AI tools for predicting and assessing disease risks and progression. In this paper, we propose an adversaryaware multimodal convolutional autoencoder (MCAE) model for cancer susceptibility prediction from multi-omics data consisting of copy number variations (CNVs), miRNA expression, and gene expression (GE). Based on different representational learning techniques, the MCAE model learns multimodal feature representations from multi-omics data, followed by classifying the patient cohorts into different cancer types on multimodal embedding space that exhibit similar characteristics in end-to-end setting. To make the MCAE model robust to adversaries and to provide consistent diagnosis, we formulate robustness as a property, such that predictions remain stable with regard to small variations in the input. We study different adversarial attacks scenarios and take both proactive and reactive measures (e.g., adversarial retraining and identification of adversarial inputs). Experiment results show that the MCAE model based on latent representation concatenation (LRC) exhibits high confidence at predicting cancer types, giving an average precision and Matthews correlation coefficient (MCC) scores of 0.9625 and 0.8453, respectively and shows higher robustness when compared and tested with state-of-the-art approaches against different attack scenarios w.r.t. ERM and CLEVER scores. Overall, our study suggests that a well-fitted and adversarially robust model can provide consistent and reliable diagnosis for cancer.

show abstract

Multimodal Autoencoder Predicts fNIRS Resting State From EEG Signals

Cited by 22 publications

References 126 publications

Error-related brain state analysis using electroencephalography in conjunction with functional near-infrared spectroscopy during a complex surgical motor task

Error-related brain state analysis using electroencephalography in conjunction with functional near-infrared spectroscopy during a complex surgical motor task

Deep learning in fNIRS: a review

Adversary-Aware Multimodal Neural Networks for Cancer Susceptibility Prediction From Multiomics Data

Contact Info

Product

Resources

About