Intermodal timing relations and audio-visual speech recognition by normal-hearing adults

Abstract

Audio-visual identification of sentences was measured as a function of audio delay in untrained observers with normal hearing; the soundtrack was replaced by rectangular pulses originally synchronized to the closing of the talker's vocal folds and then subjected to delay. When the soundtrack was delayed by 160 ms, identification scores were no better than when no acoustical information at all was provided. Delays of up to 80 ms had little effect on group-mean performance, but a separate analysis of a subgroup of better lipreaders showed a significant trend of reduced scores with increased delay in the range from 0-80 ms. A second experiment tested the interpretation that, although the main disruptive effect of the delay occurred on a syllabic time scale, better lipreaders might be attempting to use intermodal timing cues at a phonemic level. Normal-hearing observers determined whether a 120-Hz complex tone started before or after the opening of a pair of liplike Lissajou figures. Group-mean difference limens (70.7% correct DLs) were - 79 ms (sound leading) and + 138 ms (sound lagging), with no significant correlation between DLs and sentence lipreading scores. It was concluded that most observers, whether good lipreaders or not, possess insufficient sensitivity to intermodal timing cues in audio-visual speech for them to be used analogously to voice onset time in auditory speech perception. The results of both experiments imply that delays of up to about 40 ms introduced by signal-processing algorithms in aids to lipreading should not materially affect audio-visual speech understanding.

Audiovisual robustness: exploring perceptual tolerance to asynchrony and quality distortion

“…Like previous works have shown [21,26,42], asynchrony can be detected at very short thresholds, likely depending on the nature of the audiovisual event. Our observations indicate that such caution would involve the avoidance of video stream delays entirely, since the windows of temporal integration for two of three content types do not extend to audio lead asynchrony, irrespective of distortion.…”

“…While some experiments implement a temporal order judgement task, asking participants to determine which signal precedes the other [39], others rely on a simultaneity judgement task that requires assessments on perceived synchrony [5]. Still others ask for the detection of gradually introduced asynchrony [8], or discrimination between presentations with different temporal offsets [26]. Figure 1 provides a summary of previously published thresholds corresponding to perceived synchrony for different event types, but also established using different measures.…”

“…1 Overview of previously published thresholds of perceived synchrony, illustrating the different measures and event types applied to the study of temporal integration. Where some ask participants to indicate the point at which an auditory and a visual stream are no longer perceived as synchronous [8,14,21,26], others use simultaneity judgements and curve fittings to calculate the full-width at half-maximum (FWHM) [5] or the 95 % confidence interval [42]. Others still use temporal order judgements and calculate the slopes of cumulative distribution functions [11] With new multimedia platforms come new approaches to efficient streaming, and with them come old concerns.…”

Visual influences on auditory pluck and bow judgments

“…Thus, the video token of the original tape and an edited audio token were recorded simultaneously onto a second tape, resulting in a new synchronous audiovisual token. The lag time for dubbing was found to be no greater than 9.4 msec, well below the 80-msec range required for observers to detect an audiovisual asynchrony (McGrath & Summerfield, 1985).…”

Perception of intersensory synchrony: A tutorial review

“…In the McGurk effect, a lipread speech token affects the phonetic content of a speech sound that is heard (McGurk & MacDonald, 1976). The effect was biggest when auditory vowels were synchronized with the original mouth movements (McGrath & Summerfield, 1985), but the effect survives, even if audition lagged vision by 180 msec (see also Soto-Faraco & Alsius, 2007; these studies show that participants can still perceive a McGurk effect when they can quite reliably perform TOJs at intervals above the JND). So, speech-sound identification can be influenced by lipread speech even if the two are perceived as being out of sync.…”