Learning to recognize unfamiliar talkers: Listeners rapidly form representations of facial dynamic signatures

Jesse, Alexandra; Bartoli, Michael

doi:10.1016/j.cognition.2018.03.018

Cited by 6 publications

(20 citation statements)

References 97 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These aftereffects indicated a shift in the visual phonetic categories to include the ambiguous speech gesture into the category intended by the speaker. These results fit within a larger literature showing that perceivers are sensitive to visual idiosyncrasies (Heald & Nusbaum, 2014;Yakel, Rosenblum, & Fortier, 2000) and learn about them (Jesse & Bartoli, 2018;van der Zande, Jesse, & Cutler, 2013). Audiovisual speech thus allows listeners to perceive speech more reliably, in that it disambiguates the currently experienced speech but also facilitates future speech perception by recalibrating phonetic categories so that listeners can accommodate to a talker.…”

supporting

confidence: 85%

“…Lexical information, however, recalibrates visual phonetic categories directly, and not indirectly via the recalibration of auditory categories (van der Zande et al, 2013). Listeners also use talkers' idiosyncratic realizations of visual speech to form representations of these talkers' identities (Jesse & Bartoli, 2018). These representations allow listeners to recognize talkers even from new utterances.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

Attentional resources contribute to the perceptual learning of talker idiosyncrasies in audiovisual speech

Jesse

Kaplan

2019

Atten Percept Psychophys

Self Cite

View full text Add to dashboard Cite

To recognize audiovisual speech, listeners evaluate and combine information obtained from the auditory and visual modalities. Listeners also use information from one modality to adjust their phonetic categories to a talker's idiosyncrasy encountered in the other modality. In this study, we examined whether the outcome of this cross-modal recalibration relies on attentional resources. In a standard recalibration experiment in Experiment 1, participants heard an ambiguous sound, disambiguated by the accompanying visual speech as either /p/ or /t/. Participants' primary task was to attend to the audiovisual speech while either monitoring a tone sequence for a target tone or ignoring the tones. Listeners subsequently categorized the steps of an auditory /p/-/t/ continuum more often in line with their exposure. The aftereffect of phonetic recalibration was reduced, but not eliminated, by attentional load during exposure. In Experiment 2, participants saw an ambiguous visual speech gesture that was disambiguated auditorily as either /p/ or /t/. At test, listeners categorized the steps of a visual /p/-/t/ continuum more often in line with the prior exposure. Imposing load in the auditory modality during exposure did not reduce the aftereffect of this type of cross-modal phonetic recalibration. Together, these results suggest that auditory attentional resources are needed for the processing of auditory speech and/or for the shifting of auditory phonetic category boundaries. Listeners thus need to dedicate attentional resources in order to accommodate talker idiosyncrasies in audiovisual speech.

show abstract

supporting

confidence: 85%

Section: Discussionmentioning

confidence: 99%

Attentional resources contribute to the perceptual learning of talker idiosyncrasies in audiovisual speech

Jesse

Kaplan

2019

Atten Percept Psychophys

Self Cite

View full text Add to dashboard Cite

show abstract

“…Visual interference and/or "McGurk" illusory effects (fusion of conflicting auditory and visual phonetic content; McGurk & MacDonald, 1976) are observed with point-light stimuli (Rosenblum & Saldaña, 1996), when visual detail is removed by severe blurring or spatial quantization (MacDonald, Andersen, & Bachmann, 2000;Thomas & Jordan, 2002), and when visual speech is observed from distances up to 20 meters (Jordan & Sergeant, 2000) or in nonfoveal regions of the visual field (Paré, Richler, ten Hove, & Munhall, 2003). Moreover, observers can identify particular talkers -and even learn the identity of particular talkers -from dynamic visual speech information alone (Girges, Spencer, & O'Brien, 2015;Jesse & Bartoli, 2018;Rosenblum, Niehus, & Smith, 2007;Rosenblum, Smith, Nichols, Hale, & Lee, 2006;Rosenblum et al, 2002), and the natural movement of a talker's head alone provides a significant boost to audiovisual speech recognition in noise (K. G. .…”

Section: The Role Of Multisensory Temporal Covariation In Audiovisualmentioning

confidence: 99%

The Role of Multisensory Temporal Covariation in Audiovisual Speech Recognition in Noise

Jh¹,

Sandlin²,

Wojno³

et al. 2019

Preprint

View full text Add to dashboard Cite

Static and dynamic visual speech cues contribute to audiovisual (AV) speech recognition in noise. Static cues (e.g., “lipreading”) provide complementary information that enables perceivers to ascertain ambiguous acoustic-phonetic content. The role of dynamic cues is less clear, but one suggestion is that temporal covariation between facial motion trajectories and the speech envelope enables perceivers to recover a more robust representation of the time-varying acoustic signal. Modeling studies show this is computationally feasible, though it has not been confirmed experimentally. We conducted two experiments to determine whether AV speech recognition depends on the magnitude of cross-sensory temporal coherence (AVC). In Experiment 1, sentence-keyword recognition in steady-state noise (SSN) was assessed across a range of signal-to-noise ratios (SNRs) for auditory and AV speech. The auditory signal was unprocessed or filtered to remove 3-7 Hz temporal modulations. Filtering severely reduced AVC (magnitude-squared coherence of lip trajectories with cochlear-narrowband speech envelopes), but did not reduce the magnitude of the AV advantage (AV > A; ~ 4 dB). This did not depend on the presence of static cues, manipulated via facial blurring. Experiment 2 assessed AV speech recognition in SSN at a fixed SNR (-10.5 dB) for subsets of Exp. 1 stimuli with naturally high or low AVC. A small effect (~ 5% correct; high-AVC > low-AVC) was observed. A computational model of AV speech intelligibility based on AVC yielded good overall predictions of performance, but over-predicted the differential effects of AVC. These results suggest the role and/or computational characterization of AVC must be re-conceptualized.

show abstract

“…Subsequent experiments from this study also revealed that this sort of talker learning could be extended to the more difficult task of identifying four talkers. Furthermore, these researchers found that their talker learning effects were not driven by differences in talker sex, (Jesse & Bartoli, 2018;Jesse & Saba 2017). These results are complementary to the sinewave speech results of Remez and his colleagues (i.e., Sheffert et al, 2002) who (as noted above) showed that unfamiliar talkers could be learned and subsequently recognized in sinewave speech.…”

Section: Introductionmentioning

confidence: 98%

“…More recent research with point-light speech now demonstrates that these stimuli can also support talker learning (Jesse & Bartoli, 2018). In a recent study, participants were trained to identify two novel talkers in point-light speech.…”

Section: Introductionmentioning

confidence: 99%

Cross-modal transfer of talker-identity learning

Simmons

Dorsi

Dias

et al. 2020

Atten Percept Psychophys

View full text Add to dashboard Cite

A speech signal carries information about meaning and about the talker conveying that meaning. It is now known that these two dimensions are related. There is evidence that gaining experience with a particular talker in one modality not only facilitates better phonetic perception in that modality, but also transfers across modalities to allow better phonetic perception in the other. This finding suggests that experience with a talker provides familiarity with some amodal properties of their articulation such that the experience can be shared across modalities. The present study investigates if experience with talker-specific articulatory information can also support cross-modal talker learning. In Experiment 1 we show that participants can learn to identify ten novel talkers from point-light and sinewave speech, expanding on prior work. Point-light and sinewave speech also supported similar talker identification accuracies, and similar patterns of talker confusions were found across stimulus types. Experiment 2 showed these stimuli could also support cross-modal talker matching, further expanding on prior work. Finally, in Experiment 3 we show that learning to identify talkers in one modality (visual-only point-light speech) facilitates learning of those same talkers in another modality (auditory-only sinewave speech). These results suggest that some of the information for talker identity takes a modality-independent form.

show abstract

Learning to recognize unfamiliar talkers: Listeners rapidly form representations of facial dynamic signatures

Cited by 6 publications

References 97 publications

Attentional resources contribute to the perceptual learning of talker idiosyncrasies in audiovisual speech

Attentional resources contribute to the perceptual learning of talker idiosyncrasies in audiovisual speech

The Role of Multisensory Temporal Covariation in Audiovisual Speech Recognition in Noise

Cross-modal transfer of talker-identity learning

Contact Info

Product

Resources

About