A Swallowing Decoder Based on Deep Transfer Learning: AlexNet Classification of the Intracranial Electrocorticogram

Abstract

To realize a brain–machine interface to assist swallowing, neural signal decoding is indispensable. Eight participants with temporal-lobe intracranial electrode implants for epilepsy were asked to swallow during electrocorticogram (ECoG) recording. Raw ECoG signals or certain frequency bands of the ECoG power were converted into images whose vertical axis was electrode number and whose horizontal axis was time in milliseconds, which were used as training data. These data were classified with four labels (Rest, Mouth open, Water injection, and Swallowing). Deep transfer learning was carried out using AlexNet, and power in the high-[Formula: see text] band (75–150[Formula: see text]Hz) was the training set. Accuracy reached 74.01%, sensitivity reached 82.51%, and specificity reached 95.38%. However, using the raw ECoG signals, the accuracy obtained was 76.95%, comparable to that of the high-[Formula: see text] power. We demonstrated that a version of AlexNet pre-trained with visually meaningful images can be used for transfer learning of visually meaningless images made up of ECoG signals. Moreover, we could achieve high decoding accuracy using the raw ECoG signals, allowing us to dispense with the conventional extraction of high-[Formula: see text] power. Thus, the images derived from the raw ECoG signals were equivalent to those derived from the high-[Formula: see text] band for transfer deep learning.

Motor and sensory cortical processing of neural oscillatory activities revealed by human swallowing using intracranial electrodes

“…Some studies regarding the above neural oscillations were investigated for clinical applications. The lowerfrequency neural oscillations in the sensorimotor cortex measured by scalp electroencephalogram (EEG) were used for decoding in rehabilitation using brain-machine interface techniques (Ramos-Murguialday et al, 2013;Shindo et al, 2011), whereas HG activities could be used to decode more accurate, rather than lower, frequency bands (Hashimoto et al, 2020a;Yanagisawa et al, 2011Yanagisawa et al, , 2012a. PAC could be used to detect or predict epileptic seizures (Amiri et al, 2019;Edakawa et al, 2016).…”

Swallowing‐related neural oscillation: an intracranial EEG study

“…High γ activity is a key oscillation that reflects the neural processing of sensory, motor, and cognitive events 23–25 . High γ activities also show better functional localization than that in lower frequency bands 23 and have come to be known as useful features for decoding neural signals 12 . Here, we hypothesized that the analysis of high γ activities acquired by ECoG measurements would be able to reveal swallowing‐related cerebral oscillation changes in the order of milliseconds that have remained unclear.…”

“…11 Next, we demonstrated that deep transfer learning applied to intracranial electroencephalogram (EEG) signals was able to classify swallowing intention well. 12 Current neuromodulation strategies for dysphagia stimulate brain areas or peripheral organs regardless of patients' swallowing intention. We hypothesized that if the stimulation was executed by the patients' swallowing intention, a higher degree of recovery would be obtained by sensory neurofeedback.…”

“…[23][24][25] High c activities also show better functional localization than that in lower frequency bands 23 and have come to be known as useful features for decoding neural signals. 12 Here, we hypothesized that the analysis of high c activities acquired by ECoG measurements would be able to reveal swallowingrelated cerebral oscillation changes in the order of milliseconds that have remained unclear. Novel findings may provide neuromodulation strategies for dysphagia with beneficial information, such as brain areas and timing for stimulation, that will enable us to achieve better results.…”

“…First, we developed a novel swallow‐tracking system that enabled us to noninvasively monitor swallowing in real time 11 . Next, we demonstrated that deep transfer learning applied to intracranial electroencephalogram (EEG) signals was able to classify swallowing intention well 12 . Current neuromodulation strategies for dysphagia stimulate brain areas or peripheral organs regardless of patients’ swallowing intention.…”