How Optimal is Word-Referent Identification Under Multimodal Uncertainty?

Abstract

Identifying a spoken word in a referential context requires both the ability to integrate multimodal input and the ability to reason under uncertainty. How do these tasks interact with one another? We study how adults identify novel words under joint uncertainty in the auditory and visual modalities and we propose an ideal observer model of how cues in these modalities are combined optimally. Model predictions are tested in four experiments where recognition is made under various sources of uncertainty. We found that participants use both auditory and visual cues to recognize novel words. When the signal is not distorted with environmental noise, participants weight the auditory and visual cues optimally, that is, according to the relative reliability of each modality. In contrast, when one modality has noise added to it, human perceivers systematically prefer the unperturbed modality to a greater extent than the optimal model does. This work extends the literature on perceptual cue combination to the case of word recognition in a referential context. In addition, this context offers a link to the study of multimodal information in word meaning learning.

Children flexibly seek visual information to support signed and spoken language comprehension.

“…Consider a speaker who asks you to "Pass the salt" in a noisy restaurant where it is difficult to perceive what she is saying. Recent theoretical and empirical work suggests that children and adults can overcome a "noisy signal" by integrating what they perceive with their prior beliefs about what the speaker was likely to have said (Fourtassi & Frank, 2017;Gibson, Bergen, & Piantadosi, 2013;Yurovsky, Case, & Frank, 2017). In the current work, we pursue this idea and ask whether listeners strategically gather visual information to integrate with the linguistic signal and facilitate comprehension.…”

Integration of gaze information during online language comprehension and learning

“…For example, in prior work, we found that children and adults fixated longer on a speaker's face when processing familiar words in a "noisy" auditory environment, suggesting that they compensated for uncertainty in the auditory signal by gathering more visual information (MacDonald, Marchman, Fernald, & Frank, 2018). Moreover, recent theoretical and empirical work suggests that children and adults handle noise in the signal by integrating what they perceive with their prior beliefs about the speaker's intended meaning (Fourtassi & Frank, 2017;Gibson, Bergen, & Piantadosi, 2013;Yurovsky, Case, & Frank, 2017).…”

Children flexibly seek visual information during signed and spoken language comprehension

“…Recent theoretical and empirical work suggests that children and adults handle this sort of noise in the signal by integrating what they perceive with their prior beliefs about the speaker's intended meaning (Fourtassi & Frank, 2017;Gibson, Bergen, & Piantadosi, 2013;Yurovsky, Case, & Frank, 2017).…”