Computation-Based Feature Representation of Body Expressions in the Human Brain

Abstract

Humans and other primate species are experts at recognizing body expressions. To understand the underlying perceptual mechanisms, we computed postural and kinematic features from affective whole-body movement videos and related them to brain processes. Using representational similarity and multivoxel pattern analyses, we showed systematic relations between computation-based body features and brain activity. Our results revealed that postural rather than kinematic features reflect the affective category of the body movements. The feature limb contraction showed a central contribution in fearful body expression perception, differentially represented in action observation, motor preparation, and affect coding regions, including the amygdala. The posterior superior temporal sulcus differentiated fearful from other affective categories using limb contraction rather than kinematics. The extrastriate body area and fusiform body area also showed greater tuning to postural features. The discovery of midlevel body feature encoding in the brain moves affective neuroscience beyond research on high-level emotion representations and provides insights in the perceptual features that possibly drive automatic emotion perception.

A computational neuroethology perspective on body and expression perception

“…In another study, we showed that the midlevel features 'limb contraction' and 'limb angles' play a central role in fearful body expression perception and are specifically represented in action observation, motor preparation, and affect coding regions, including the amygdala (Figure 3) [9]. Importantly, with this new feature-based approach we can now clarify in detail the functions of areas that have so far only been associated with general body selectivity at the visual category level.…”

“…Midlevel features are different from classical low-level visual features (e.g., edges, spatial frequency, motion direction) [7] as well as from subjective semantic features that we intuitively notice and believe to be the features we act upon (i.e., high-level semantic categories of emotions, actions, and intentions) [8]. Some examples of midlevel feature candidates derived from computational analysis of body posture and movements are limb contraction [3,9], head orientation, and hand to head distance [10]. Recent studies identified brain correlates of semantic features such as agentic action [11], animacy [12], and sociality [13].…”

“…For example, the action observation network shows increased activity for threatening body expressions than for neutral expressions [27,32,33]. The motor system, that is responsible for action preparation, also plays an important role, especially in the case of fear [31,[34][35][36][37], as do subcortical areas [9,38] and cerebellum [39]. A subcortical pathway between the pulvinar, superior colliculus, and amygdala interacts with other areas to support defensive reflexes (e.g., withdrawal, freezing, startle) [40,41], and specifically does so for threatening body expressions [28].…”

“…Feature-based approach to body expression perception. Poyo Solanas et al [9] investigated the subcategorical representation of emotional body movements by relating computed features of body movement to multivariate patterns of fMRI data using representational similarity analysis. This figure shows the results for the feature limb contraction (average of the distances between the wrists and ankles to the head).…”

“…Abbreviations: ACC, anterior cingulate cortex; AMYG, amygdala; IPL, inferior parietal lobule; MTG, middle temporal gyrus; pIPS, intraparietal sulcus, posterior segment; PMv, premotor cortex, ventral part; RSA, representational similarity analysis; SFG, superior frontal gyrus. Figure modified, with permission, from [9]. there is a need for methods that describe and quantify the natural behavior of freely moving humans under various natural circumstances [65,66].…”