Specification of cross-modal source information in isolated kinematic displays of speech

Abstract

Information about the acoustic properties of a talker's voice is available in optical displays of speech, and vice versa, as evidenced by perceivers' ability to match faces and voices based on vocal identity. The present investigation used point-light displays (PLDs) of visual speech and sinewave replicas of auditory speech in a cross-modal matching task to assess perceivers' ability to match faces and voices under conditions when only isolated kinematic information about vocal tract articulation was available. These stimuli were also used in a word recognition experiment under auditory-alone and audiovisual conditions. The results showed that isolated kinematic displays provide enough information to match the source of an utterance across sensory modalities. Furthermore, isolated kinematic displays can be integrated to yield better word recognition performance under audiovisual conditions than under auditory-alone conditions. The results are discussed in terms of their implications for describing the nature of speech information and current theories of speech perception and spoken word recognition.

Hearing a face: Cross-modal speaker matching using isolated visible speech

“…In fact, initial support for this prediction has been provided by four very recent articles reporting research conducted in parallel with our own study (discussed below; Kamachi, Hill, Lander, & Vatikiotis-Bateson, 2003;Lachs & Pisoni, 2004a, 2004b, 2004c. These four articles reported that, with an XAB methodology, observers were able to successfully match voices to faces and faces to voices.…”

“…These auditory tests involved either deleting the phonetically extraneous information from the signal (e.g., sine wave speech; Kamachi et al, 2003;Lachs & Pisoni, 2004b, 2004c or testing whether acoustic distortions that hinder phonetic perception also hinder cross-modal matching (Lachs & Pisoni, 2004a). The results of these tests support the conclusion that the idiosyncratic phonetic information contained in the auditory signal is salient for cross-modal matching.…”

“…They proposed that in both auditory and visible speech signals, there is speaker-specific articulatory information that specifies both phonetic message and speaker (Kamachi et al, 2003;Lachs & Pisoni, 2004a, 2004b, 2004c. It was thought that this articulatory information conveys an idiosyncratic speaking style that can be specified in the timevarying dimensions of both auditory and visual signals (see also Rosenblum, 2004Rosenblum, , 2005.…”

“…With regard to the salience of visible speaking style for cross-modal matching, three of the four studies (Kamachi et al, 2003;Lachs & Pisoni, 2004b, 2004c reported experiments designed to test whether visible articulatory movements are important for these purposes. In normal speaking face displays, there is necessarily more information than just articulatory movement information available that could support matching judgments.…”

“…Thus, it can be argued that these studies failed to thoroughly establish whether visible speech movements play a critical role in cross-modal speaker matching. Another recent project (Lachs & Pisoni, 2004c) implemented a point-light methodology to examine the importance of visible speech movements for cross-modal speaker matching. Lachs and Pisoni (2004c) applied 28 point-lights to the faces of 4 female speakers.…”