Brain-to-brain hyperclassification reveals action-specific motor mapping of observed actions in humans

Abstract

Seeing an action may activate the corresponding action motor code in the observer. It remains unresolved whether seeing and performing an action activates similar action-specific motor codes in the observer and the actor. We used novel hyperclassification approach to reveal shared brain activation signatures of action execution and observation in interacting human subjects. In the first experiment, two "actors" performed four types of hand actions while their haemodynamic brain activations were measured with 3-T functional magnetic resonance imaging (fMRI). The actions were videotaped and shown to 15 "observers" during a second fMRI experiment. Eleven observers saw the videos of one actor, and the remaining four observers saw the videos of the other actor. In a control fMRI experiment, one of the actors performed actions with closed eyes, and five new observers viewed these actions. Bayesian canonical correlation analysis was applied to functionally realign observers' and actors' fMRI data. Hyperclassification of the seen actions was performed with Bayesian logistic regression trained on actors' data and tested with observers' data. Without the functional realignment, between-subjects accuracy was at chance level. With the realignment, the accuracy increased on average by 15 percentage points, exceeding both the chance level and the accuracy without functional realignment. The highest accuracies were observed in occipital, parietal and premotor cortices. Hyperclassification exceeded chance level also when the actor did not see her own actions. We conclude that the functional brain activation signatures underlying action execution and observation are partly shared, yet these activation signatures may be anatomically misaligned across individuals.

Statistical Pattern Recognition Reveals Shared Neural Signatures for Displaying and Recognizing Specific Facial Expressions

“…Yet, pattern classification work on facial expression recognition (e.g. Said et al 2010) and “hyperclassification” of seen and executed hand actions (Smirnov et al, 2017) suggests that our results should be generalizable.…”

Imaging Real-Time Tactile Interaction With Two-Person Dual-Coil fMRI

“…During the interaction, sensory, cognitive, and emotional information is constantly remapped in the observers' brain and used for motor actions as responses attuned to the received input (1). Thus the interlocutors' minds are intertwined into a shared system facilitating reciprocation (2)(3)(4) as well as anticipation of the other person's acts, allowing distribution of neural processing across brains to aid, for example, problem solving.…”

“…The communicative information can then be presented to the receiver subjects as stimuli during brain imaging, allowing joint analysis of the brain activity of the sender and receiver subjects. This line of work has revealed how successful communication via speech (25,26), hand gestures (3,27), and facial expressions (28) enhances similarity of neural activation patterns across the interlocutors in a task-specific manner. This approach however lacks any interactivity, as the receiver subjects are essentially viewing pre-recorded stimuli, and need not to generate any responses to them.…”

Imaging real-time tactile interaction with two-person dual-coil fMRI

“…Thus the interlocutors' minds are intertwined into a shared system facilitating reciprocation (Hasson et al, 2012;Nummenmaa et al, 2018;Smirnov et al, 2017) as well as anticipation of the other person's acts, allowing distribution of neural processing across brains to aid, for example, problem solving.…”

“…The communicative information can then be presented to the receiver subjects as stimuli during brain imaging, allowing joint analysis of the brain activity of the sender and receiver subjects. This line of work has revealed how successful communication via speech (Smirnov et al, 2019;Stephens et al, 2010), hand gestures (Schippers et al, 2010;Smirnov et al, 2017) and facial expressions (Anders et al, 2011) enhances similarity of neural activation patterns across the interlocutors in a task-specific manner. This approach however lacks any interactivity, as the receiver subjects are essentially viewing pre-recorded stimuli, and need not to generate any responses to them.…”