Integration of Partial Information Within and Across Modalities: Contributions to Spoken and Written Sentence Recognition

Abstract

Purpose: This study evaluated the extent to which partial spoken or written information facilitates sentence recognition under degraded unimodal and multimodal conditions. Method: Twenty young adults with typical hearing completed sentence recognition tasks in unimodal and multimodal conditions across 3 proportions of preservation. In the unimodal condition, performance was examined when only interrupted text or interrupted speech stimuli were available. In the multimodal condition, performance was examined when both interrupted text and interrupted speech stimuli were concurrently presented. Sentence recognition scores were obtained from simultaneous and delayed response conditions. Results: Significantly better performance was obtained for unimodal speech-only compared with text-only conditions across all proportions preserved. The multimodal condition revealed better performance when responses were delayed. During simultaneous responses, participants received equal benefit from speech information when the text was moderately and significantly degraded. The benefit from text in degraded auditory environments occurred only when speech was highly degraded. Conclusions: The speech signal, compared with text, is robust against degradation likely due to its continuous, versus discrete, features. Allowing time for offline linguistic processing is beneficial for the recognition of partial sensory information in unimodal and multimodal conditions. Despite the perceptual differences between the 2 modalities, the results highlight the utility of multimodal speech + text signals.

Integration of partial information for spoken and written sentence recognition by older listeners

“…In the multimodal conditions, we examined how performance is affected by response timing. We compared the results to a recent study on analogous conditions with younger normal-hearing adults (Smith and Fogerty, 2015). These findings provide insight into how older listeners resolve partially degraded written and spoken information and how multimodal processing of these signals changes with age.…”

“…Unimodal processing of partial speech and text information has been investigated in both younger and older normal-hearing listeners. Smith and Fogerty (2015) found, for younger normal-hearing adult listeners, significantly better sentence recognition for unimodal speech-only compared to text-only conditions across various proportions of signal preservation. While a direct comparison of speech versus text processing for older listeners has not been conducted to our knowledge, studies have compared performance between younger and older listeners, and revealed generally poorer performance by older listeners resolving partial speech and text information (e.g., Krull et al, 2013).…”

“…1 [leftmost plots in (A) and (B)]. A younger, normal-hearing adult group (Smith and Fogerty, 2015) is included on Fig. 1 for comparison.…”

“…Supplementing speech recognition with visual cues (or reading with auditory cues) can improve sentence intelligibility and serve in an additive manner (Grant and Seitz, 2000;Zekveld et al, 2008). Smith and Fogerty (2015) found that young normal-hearing listeners demonstrated better sentence recognition in multimodal conditions when modalities were combined compared to unimodal performance of either speech-only or text-only. This performance was also dependent on the timing allowed for response, with performance better for the delayed compared to simultaneous response (i.e., responding after or during the presented stimulus, respectively).…”

“…Speech and text stimuli were identical to Smith and Fogerty (2015). These materials were selected from recordings of a male talker (Loizou, 2007) speaking the IEEE/ Harvard sentence corpus (Rothauser et al, 1969) that contains meaningful sentences with five keywords.…”