The multimodal nature of spoken word processing in the visual world: Testing the predictions of alternative models of multimodal integration

Abstract

a b s t r a c tAmbiguity in natural language is ubiquitous, yet spoken communication is effective due to integration of information carried in the speech signal with information available in the surrounding multimodal landscape. Language mediated visual attention requires visual and linguistic information integration and has thus been used to examine properties of the architecture supporting multimodal processing during spoken language comprehension. In this paper we test predictions generated by alternative models of this multimodal system. A model (TRACE) in which multimodal information is combined at the point of the lexical representations of words generated predictions of a stronger effect of phonological rhyme relative to semantic and visual information on gaze behaviour, whereas a model in which sub-lexical information can interact across modalities (MIM) predicted a greater influence of visual and semantic information, compared to phonological rhyme. Two visual world experiments designed to test these predictions offer support for sub-lexical multimodal interaction during online language processing.

Complex Word Recognition Behaviour Emerges from the Richness of the Word Learning Environment

“…These data indicated strong activation of the semantic properties of both the semantically related object and the spoken target word but little activation of the semantic properties of the visually related object, the phonologically related object and the unrelated object. This contrasts with simulations in Smith et al 27 using a fine-coding of the phonological input which showed early strong activation of the semantic properties of the phonological competitor. Fig.…”

“…After 750,000 trials, the effect was earlier, from time step 12, all |t| ≥ 2.513, all p ≤ .040, maximum ratio = 1.667, and after 1,000,000 trials, it continued to be significant from time step 12, all |t| ≥ 2.630, all p ≤ .034, with maximum ratio = 1.525. Interestingly, as for the simulations in Smith et al 27 using a fine-coding of the phonological input, the model demonstrated a prolonged effect of phonological relatedness, which was consistent with the developmental data, but not with adult performance for such visual world tasks. Furthermore, as in the developmental behavioural data, the phonological effect was substantially smaller than that of the semantic effect (as seen in Figure 2, panels B, C and D).…”

“…For the phonological representation, this was encoded at the syllable level, so each word was represented as a unique pattern over a set of 60 units, with each unit active with probability 0.5. This differed from simulations of the model where phonology was encoded at the phoneme level, with a unique pattern distributed over 10 units representing each phoneme and each word encoded by a unique combination of 6 phonemes in the input 27 . Thus, the current simulations were a coarser-coding of the auditory input than fine-grained, phoneme-level representations that have been used to simulate word processing in participants who have phoneme-level awareness of speech stimuli.…”

“…Thus far, participants with lowliteracy have not been tested on the relative role of visual processing in the visual world paradigm and so the current simulations provide also predictions for future behavioural investigations. The model is similar to that employed in Smith et al 27 , and illustrated in Figure 1. The model comprised several modalities connecting to a set of 400 hidden units in an integrative layer: a set of 80 visual units, with 20 units each associated with one of four object positions in the visual environment; a set of 60 phonological units representing the spoken form of a monosyllabic word; and 200 semantic units, to represent the word's meaning.…”

“…The developmental trajectory of the same multimodal model's processing was also investigated 27 . In this simulation, we investigated the simultaneous involvement of phonological, semantic, and visual information in auditory word processing.…”