Impact of Different Speech Types on Listening Effort

Abstract

Listeners are exposed to different types of speech in everyday life, from natural speech to speech that has undergone modifications or has been generated synthetically. While many studies have focused on measuring the intelligibility of these distinct speech types, their impact on listening effort is not known. The current study combined an objective measure of intelligibility, a physiological measure of listening effort (pupil size) and listeners' subjective judgements, to examine the impact of four speech types: plain (natural) speech, speech produced in noise (Lombard speech), speech enhanced to promote intelligibility, and synthetic speech. For each speech type, listeners responded to sentences presented in one of three levels of speech-shaped noise. Subjective effort ratings and intelligibility scores showed an inverse ranking across speech types, with synthetic speech being the most demanding and enhanced speech the least. Pupil size measures indicated an increase in listening effort with decreasing signal-to-noise ratio for all speech types apart from synthetic speech, which required significantly more effort at the most favourable noise level. Naturally and artificially modified speech were less effortful than plain speech at the more adverse noise levels. These outcomes indicate a clear impact of speech type on the cognitive demands required for comprehension.

Using Pupil Dilation to Measure Cognitive Load When Listening to Text-to-Speech in Quiet and in Noise

“…All stimuli were mixed with speech-shaped noise at signal-tonoise ratios (SNRs) -1dB and -3dB, chosen such that the cognitive effort is increased whilst intelligibility remains close to ceiling. In accordance with the estimated psychometric function in [10] [1], the TTS condition at -5dB SNR was too difficult. The procedure described in [2] was followed exactly in terms of structure, presentation and data collection.…”

“…Although 0% WER is expected in quiet conditions, with synthetic speech of poor quality this isn't feasible: too much data would be discarded under such a strict criterion. The improvements in analysis explained in the previous section were made 1 . Two participants were excluded from the analysis as more than 50% of their trials were discarded during preprocessing.…”

Measuring the contribution to cognitive load of each predicted vocoder speech parameter in DNN-based speech synthesis

“…More interestingly, we observe that Systems D and F evoke a lower mean pupil dilation in this condition compared to the easier noise condition (-3dB). A similar finding was observed in [1] when measuring cognitive load of TTS at -5dB. It was said to indicate a task that is too challenging for the listener.…”

“…Evaluation methods generally fail to consider the cognitive load imposed by listening to synthetic speech. This is especially concerning as synthetic speech demands the greatest effort even at favourable signal-to-noise ratios [1]. This highlights the negative impact synthetic speech potentially has on the human cognitive processing system.…”

“…For low quality systems such as HMM-based speech synthesis ceiling listening effort appears to be reached already at easier SNR levels. An HMM system was used in [1] which explains the great effort demanded compared to all other speech types. Recently, HMM-based speech synthesis has been, in most cases, replaced by Deep Neural Networks (DNNs).…”