Ensemble representation for multiple facial expressions: Evidence for a capacity limited perceptual process

Abstract

We tested the processing capacity of establishing ensemble representation for multiple facial expressions using the simultaneous-sequential paradigm. Each set consisted of 16 faces conveying a variable amount of happy and angry expressions. Participants judged on a continuous scale the perceived average emotion from each face set (Experiment 1). In the simultaneous condition, the 16 faces were presented concurrently; in the sequential condition, two sets, each containing eight faces, were presented successively. Results showed that judgments varied depending on the number of happy versus angry faces contained in the sets and were sensitive at the single trial level to the perceived mean emotion intensity (based on postexperiment ratings), providing evidence of a genuine mean representation rather than the mere use of a single face or enumeration. Experiments 2 and 3 replicated Experiment 1, but implemented a different response format (binary choices) and added masks following each display, respectively. Importantly, in all three experiments, performance was consistently better in the sequential than in the simultaneous condition, revealing a limited-capacity process. A set of control analyses ruled out the use of enumeration or mere subsampling by the participants to perform the task. Collectively, these results indicate that participants could readily extract mean emotion from multiple faces shown concurrently in a set, but this process is best conceived as being capacity limited.

Individual Faces Were Not Discarded During Extracting Mean Emotion Representations

“…Furthermore, stimulus variance in a set affected the number of faces involved in ensemble coding ( Ji et al, 2020 ). Ji et al ( 2018 , 2020 ) provided direct evidence suggesting that stimulus variance in a set could affect ensemble coding. The extraction of a mean representation is a capacity-limited perceptual process ( Ji et al, 2018 ), and stimulus variance in a set might be an important factor causing different relationships between the mean representation and individual representations in different studies.…”

“… Ji et al ( 2018 , 2020 ) provided direct evidence suggesting that stimulus variance in a set could affect ensemble coding. The extraction of a mean representation is a capacity-limited perceptual process ( Ji et al, 2018 ), and stimulus variance in a set might be an important factor causing different relationships between the mean representation and individual representations in different studies. Although the present study controlled the stimulus variance in a set, we did not investigate the effect of stimulus variance on the processing of the mean emotion representation.…”

“…To explain this finding, it was suggested that processing individual familiar faces requires fewer attentional resources. The extraction of a mean representation was demonstrated to be capacity limited ( Ji et al, 2018 ) similar to processing individual items in working memory. Therefore, to explore the processing mechanisms underlying the two types of representations, researchers need to consider attentional resources; however, limited research investigated how attentional resources affect the processing mechanisms of the two types of representations.…”

Can change detection succeed when change localization fails?

“…Consistent with the view that face ensemble characteristics cannot be computed in the absence of encoding individual faces, a recent study by Neumann et al (2018) showed that manipulations that reduced the fidelity of encoding of individual faces also reduced the precision of encoding ensemble characteristics (in this case, mean identity). There is considerable evidence from other studies, however, that SSRs can be computed for emotional expressions (Haberman & Whitney, 2007, 2010; Ji, Chen, Loeys, & Pourtois, 2018; Li et al, 2016; Sun & Chong, 2020) suggesting that information about second-order relations in facial features can be aggregated across multiple faces concurrently. The face inversion effect provides an interesting potential answer to this question.…”

Averaging multiple facial expressions through subsampling

“…For instance, one could wonder how many faces in a set, if not all, contribute to the perception of that crowd's mean emotion. This question is reminiscent of a current debate in the literature about the actual meaning and function of ensemble representations, and more specifically whether they offer a means to surpass traditional bottlenecks in information processing and attention selection (Alvarez, 2011;Chong & Treisman, 2005;Cohen et al, 2016;Attarha, Moore, & Vecera, 2014;Ji, Chen, Loeys, & Pourtois, 2018). If sampling only 3-4 items (this size being traditionally assumed to correspond to the upper bound for attention capacity; see Pylyshyn & Storm, 1998) or even fewer items in a set could adequately explain human observers' performance when many more items are shown in the visual field, then the claim that ensemble representations are capacity unlimited or can overcome the bottlenecks in visual perception might need to be reconsidered.…”