Explaining face-voice matching decisions: The contribution of mouth movements, stimulus effects and response biases

Abstract

Previous studies have shown that face-voice matching accuracy is more consistently above chance for dynamic (i.e. speaking) faces than for static faces. This suggests that dynamic information can play an important role in informing matching decisions. We initially asked whether this advantage for dynamic stimuli is due to shared information across modalities that is encoded in articulatory mouth movements. Participants completed a sequential face-voice matching task with (1) static images of faces, (2) dynamic videos of faces, (3) dynamic videos where only the mouth was visible, and (4) dynamic videos where the mouth was occluded, in a well-controlled stimulus set. Surprisingly, after accounting for random variation in the data due to design choices, accuracy for all four conditions was at chance. Crucially, however, exploratory analyses revealed that participants were not responding randomly, with different patterns of response biases being apparent for different conditions. Our findings suggest that face-voice identity matching may not be possible with above-chance accuracy but that analyses of response biases can shed light upon how people attempt face-voice matching. We discuss these findings with reference to the differential functional roles for faces and voices recently proposed for multimodal person perception.

Audiovisual identity perception from naturally‐varying stimuli is driven by visual information

“…example, for a number of trait judgements (Mileva et al, 2020; but see Rezlescu et al, 2015) and independent face and voice spaces, indexing the distinctiveness of identities (Tatz et al, 2020). Similarly, studies probing cross-modal face-voice identity matching report that perceivers can determine with low but above-chance accuracy whether an unfamiliar face and voice belong to the same person or two different people, especially if the face stimulus is dynamic (Mavica & Barenholtz, 2013;Smith, Dunn, Baguley, & Stacey, 2016a;Smith, Dunn, Baguley, & Stacey, 2016b;Stevenage, Hamlin, & Ford, 2017; but see Lavan, Smith, Smith, Jiang, & McGettigan, 2021 showing chance-level performance). As a result, it has been speculated that there is overlapping identity information between the auditory and visual modality, potentially encoded in the dynamic articulatory movements (Kamachi, Hill, Lander, & Vatikiotis-Bateson, 2003;Lachs & Pisoni, 2004;Lander, Hill, Kamachi, & Vatikiotis-Bateson, 2007; but see Lavan et al, 2021) or in concordant cues for person-related judgements, such as a person's gender, age, height, or weight (Smith et al, 2016b).…”

The effects of the presence of a face and direct eye gaze on voice identity learning

“…For identity perception, this account thus strongly assumes that there is meaningful shared and complementary identity-related information across the auditory and visual modalities. While there are studies that suggest that there is limited shared information, others, however, report low or close to chance-level accuracy for any cross-modal identity matching (see Lavan, Smith, et al, 2021;Smith et al, 2016aSmith et al, , 2016b.…”