Using Pupillometry to Measure the Cognitive Load of Synthetic Speech

Govender, Avashna; King, Simon

doi:10.21437/interspeech.2018-1174

Cited by 30 publications

(35 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Different aspects of the synthetic speech appear to have an effect, depending on the type of task chosen, notably linguistic vs. non-linguistic. In ongoing work we are investigating pupillometry as a more sensitive measure of listening effort [24].…”

Section: Resultsmentioning

confidence: 99%

Measuring the Cognitive Load of Synthetic Speech Using a Dual Task Paradigm

Govender

King

2018

Interspeech 2018

Self Cite

View full text Add to dashboard Cite

We present a methodology for measuring the cognitive load (listening effort) of synthetic speech using a dual task paradigm. Cognitive load is calculated from changes in a listener's performance on a secondary task (e.g., reaction time to decide if a visually-displayed digit is odd or even). Previous related studies have only found significant differences between the best and worst quality systems but failed to separate the systems that lie in between. A paradigm that is sensitive enough to detect differences between state-of-the-art, high quality speech synthesizers would be very useful for advancing the state of the art. In our work, four speech synthesis systems from a previous Blizzard Challenge, and the corresponding natural speech, were compared. Our results show that reaction times slow down as speech quality reduces, as we expected: lower quality speech imposes a greater cognitive load, taking resources away from the secondary task. However, natural speech did not have the fastest reaction times. This intriguing result might indicate that, as speech synthesizers attain near-perfect intelligibility, this paradigm is measuring something like the listener's level of sustained attention and not listening effort.

show abstract

Section: Resultsmentioning

confidence: 99%

Measuring the Cognitive Load of Synthetic Speech Using a Dual Task Paradigm

Govender

King

2018

Interspeech 2018

Self Cite

View full text Add to dashboard Cite

show abstract

“…As in [11], the speech stimuli were played to listeners through headphones in a noise-and light-controlled room. Simultane-ously, pupil size was measured using an eye tracker.…”

Section: Experimental Set-upmentioning

confidence: 99%

“…In our previous works, we used pupillometry to measure the cognitive load of synthetic speech [11,12]. This method has proven to be reliable in showing pupil dilation as an index of listening effort under noisy conditions.…”

Section: Introductionmentioning

confidence: 99%

Measuring the contribution to cognitive load of each predicted vocoder speech parameter in DNN-based speech synthesis

Govender¹,

Valentini-Botinhao²,

King³

2019

10th ISCA Workshop on Speech Synthesis (SSW 10)

Self Cite

View full text Add to dashboard Cite

Listening to even high quality text-to-speech-such as that generated by a Deep Neural Network (DNN) driving a vocoder-still requires greater cognitive effort than natural speech, under noisy conditions. Vocoding itself, plus errors in predictions of the vocoder speech parameters by the DNN model are assumed to be responsible. To better understand the contribution of each parameter, we construct a range of systems that vary from copysynthesis (i.e., vocoding) to full text-to-speech generated using a Deep Neural Network system. Each system combines some speech parameters (e.g., spectral envelope) from copy-synthesis with other speech parameters (e.g., F0) predicted from text. Cognitive load was measured using a pupillometry paradigm described in our previous work. Our results reveal the differing contributions that each predicted speech parameter makes to increasing cognitive load.

show abstract

“…Comprehensibility of an utterance means how smoothly listeners can understand the content of that utterance, and degree of comprehensibility has been often quantified by listeners' subjective rating [2]. Listening effort [10] and cognitive load [11] seem to be strongly related to comprehensibility, i.e. smoothness of understanding.…”

Section: Introductionmentioning

confidence: 99%

“…One possible solution is collecting data related to listeners' behaviors or responses while listening to non-native speech. In [18,19,10], EEG (electroencephalogram) recordings were made from listeners and listening efforts were discussed quantitatively and in [11], eye-trackers were used to measure the size of pupils to predict the magnitude of cognitive load when listening. These features are strongly related to comprehensibility and obtained recordings may give us a sequential data of comprehensibility, but physiological sensing requires expensive devices.…”

Section: Introductionmentioning

confidence: 99%

Native Listeners' Shadowing of Non-native Utterances as Spoken Annotation Representing Comprehensibility of the Utterances

Lin

Inoue

Trisitichoke

et al. 2019

8th ISCA Workshop on Speech and Language Technology in Education (SLaTE 2019)

View full text Add to dashboard Cite

Recently, researchers' attention has been paid to pronunciation assessment not based on comparison between learners' utterances and native models, but based on comprehensibility of the utterances [1, 2, 3]. In our previous studies [4, 5], native listeners' shadowing was investigated and shown to be effective to predict comprehensibility perceived by listeners (shadowers). In this paper, native listeners' shadowings are viewed as spoken annotations that can represent comprehensibility. In [4, 5], to predict comprehensibility of a non-native utterance, the GOP scores of its corresponding native listeners' shadowings were calculated by using a DNN-based ASR front-end. Generally speaking, annotations are prepared manually and, even when some techniques are used for annotations, only stable and reliable techniques should be used. In this paper, a simpler, stabler, and more reliable method to derive comprehensibility annotations was proposed. After native listeners' shadowing, they are asked to read aloud the sentence intended by the learner. Reading is the most prepared speech and shadowing is probably the least prepared speech. DTW between the two utterances is supposed to be able to quantify and predict comprehensibility or shadowability perceived by the shadowers. In experiments, DTW between shadowings and readings shows higher correlation than the GOP scores of shadowings.

show abstract

Using Pupillometry to Measure the Cognitive Load of Synthetic Speech

Cited by 30 publications

References 11 publications

Measuring the Cognitive Load of Synthetic Speech Using a Dual Task Paradigm

Measuring the Cognitive Load of Synthetic Speech Using a Dual Task Paradigm

Measuring the contribution to cognitive load of each predicted vocoder speech parameter in DNN-based speech synthesis

Native Listeners' Shadowing of Non-native Utterances as Spoken Annotation Representing Comprehensibility of the Utterances

Contact Info

Product

Resources

About