Behavioral and Neural Variability of Naturalistic Arm Movements

Peterson, Steven M.; Singh, Satpreet H.; Wang, Nancy X. R.; Rao, Rajesh P. N.; Brunton, Bingni W.

doi:10.1523/eneuro.0007-21.2021

Cited by 8 publications

(16 citation statements)

References 103 publications

(98 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We assessed cross-modal, self-supervised decoding performance on four datasets (table 1) and demonstrate in each case that cross-modal decoding outperforms unimodal, self-supervised models and approaches the accuracy of supervised models. We consider three movement decoding tasks: determining whether a participant's arm was moving or at rest (ECoG move/rest [56,77] and EEG move/rest [73]), predicting which of five fingers was being flexed (ECoG finger flexion [40,74]), and determining whether a participant was exposed to a visual or physical balance perturbation while either walking or standing (EEG balance perturbations [65]). Our decoding models all use the HTNet architecture, a compact convolutional neural network that has been demonstrated to perform well at decoding ECoG/EEG data [25,78].…”

Section: Resultsmentioning

confidence: 99%

“…Data streams include electrocorticography (ECoG), electroencephalography (EEG), electromyography (EMG), and multiple kinematic measurements. Kinematic measurements were obtained from markerless motion capture applied to video recordings (ECoG move/rest [56]), exoskeleton positions (EEG move/rest [73]), dataglove recordings (ECoG finger flexion [74]), and motion capture markers (EEG balance perturbations [65]). We computed the number of events per participant after balancing events across classes.…”

Section: Resultsmentioning

confidence: 99%

“…Relevant features for neural decoding often lack a clear baseline [20,45], are non-stationary over long time periods [46,47], exponentially decrease in amplitude at higher frequencies [48,49], and occur in a small fraction of the total recording electrodes [50,51]. Second, while contrastive techniques have been applied for neural decoding [35,[52][53][54][55], neural data can be noisy and variable from one example to the next, so creating dissimilar examples is often difficult, even with labeled data [56,57]. Furthermore, many self-supervised approaches in computer vision augment the input images during training to improve model robustness [39,58,59].…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Learning neural decoders without labels using multiple data streams

2022

Self Cite

View full text Add to dashboard Cite

Objective: Recent advances in neural decoding have accelerated the development of brain-computer interfaces aimed at assisting users with everyday tasks such as speaking, walking, and manipulating objects. However, current approaches for training neural decoders commonly require large quantities of labeled data, which can be laborious or infeasible to obtain in real-world settings. Alternatively, self-supervised models that share self-generated pseudo-labels between two data streams have shown exceptional performance on unlabeled audio and video data, but it remains unclear how well they extend to neural decoding. Approach: We learn neural decoders without labels by leveraging multiple simultaneously recorded data streams, including neural, kinematic, and physiological signals. Speciﬁcally, we apply cross-modal, self-supervised deep clustering to train decoders that can classify movements from brain recordings. After training, we then isolate the decoders for each input data stream and compare the accuracy of decoders trained using cross-modal deep clustering against supervised and unimodal, self-supervised models. Main results: We find that sharing pseudo-labels between two data streams during training substantially increases decoding performance compared to unimodal, self-supervised models, with accuracies approaching those of supervised decoders trained on labeled data. Next, we extend cross-modal decoder training to three or more modalities, achieving state-of-the-art neural decoding accuracy that matches or slightly exceeds the performance of supervised models. Signiﬁcance: We demonstrate that cross-modal, self-supervised decoding can be applied to train neural decoders when few or no labels are available and extend the cross-modal framework to share information among three or more data streams, further improving self-supervised training.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Learning neural decoders without labels using multiple data streams

2022

Self Cite

View full text Add to dashboard Cite

show abstract

“…This analysis also excluded the first 5 s of each short clip within each video block, so that it is aligned with the ISC analysis. In general, employing a machine learning algorithm to track movements of human limbs has been found to be promising in studies of naturalistic human movements ( Peterson et al, 2021 ). For the automatic annotation of movements, we implemented the OpenPose github repository ( https://github.com/CMU-Perceptual-Computing-Lab/openpose ), as in Ntoumanis et al (2022) .…”

Section: Methodsmentioning

confidence: 99%

Developmental differences in the perception of naturalistic human movements

Ntoumanis

Шестакова

Koriakina

et al. 2023

Front. Hum. Neurosci.

View full text Add to dashboard Cite

IntroductionIt is widely believed that we are more attentive towards moving versus static stimuli. However, the neural correlates underlying the perception of human movements have not been extensively investigated in ecologically valid settings, nor has the developmental aspect of this phenomenon. Here, we set forth to investigate how human limb movements displayed in naturalistic videos influence the attentional engagement of children and young adults.MethodsThirty-nine healthy participants (4–26 years old) were presented with naturalistic videos featuring human goal-directed movements, while neural activity was recorded using electroencephalography (EEG). Video scenes were automatically annotated as containing arm, leg or no movement, using a machine learning algorithm. The viewers’ attentional engagement was quantified by the intersubject correlation of EEG responses evoked by the videos.ResultsOur results demonstrate that scenes featuring limb movements, especially simultaneous arm and leg movements, elicit higher attentional engagement than scenes with no limb movement. Interestingly, this effect was found to diminish with age.DiscussionOverall, our findings extend previous work on the perception of human motion by implementing naturalistic stimuli in the experimental design and extend the list of factors influencing the viewer’s engagement exerted by naturalistic videos.

show abstract

“…Highly invasive solutions such as intracranial EEG also may not be acceptable or justifiable for many participants. 16,46–48 Subgaleal EEG (sgEEG) allows for an ultra-long-term EEG recording with high stability and data quality. There may be applications beyond epilepsy, such as brain-computer interfaces or gaming.…”

Section: Perspective and Future Workmentioning

confidence: 99%