Interactive Hesitation Synthesis: Modelling and Evaluation

Abstract

Abstract:Conversational spoken dialogue systems that interact with the user rather than merely reading the text can be equipped with hesitations to manage dialogue flow and user attention. Based on a series of empirical studies, we elaborated a hesitation synthesis strategy for dialogue systems, which inserts hesitations of a scalable extent wherever needed in the ongoing utterance. Previously, evaluations of hesitation systems have shown that synthesis quality is affected negatively by hesitations, but that they result in improvements of interaction quality. We argue that due to its conversational nature, hesitation synthesis needs interactive evaluation rather than traditional mean opinion score (MOS)-based questionnaires. To validate this claim, we dually evaluate our system's speech synthesis component, on the one hand, linked to the dialogue system evaluation, and on the other hand, in a traditional MOS way. We are thus able to analyze and discuss differences that arise due to the evaluation methodology. Our results suggest that MOS scales are not sufficient to assess speech synthesis quality, leading to implications for future research that are discussed in this paper. Furthermore, our results indicate that synthetic hesitations are able to increase task performance and that an elaborated hesitation strategy is necessary to avoid likability issues.

Fill the silence! Basics for modeling hesitation

“…One practical application of the results obtained here is the extension of the hesitation insertion model for speech synthesis, which has been prototypically tested in Betz et al (2018), which did not yet take into account the structural interplay of silences and fillers. Furthermore, the hesitation model by Betz et al (2018) is centered on lengthening, which provides an elegant entry point for a synthetic hesitation interval, and reflects human speech production by making use of the pre-planned, but not-yet-uttered words in the articulatory buffer (Levelt, 1989). This approach receives support by the confirmed notion of longer silences after um-type fillers: the presence of a nasal sound makes this type of filler a better candidate to smoothly initiate a hesitation interval by lengthening compared to the uh-type fillers (for hesitation lengthening distribution over phone types, cf.…”

Speech Synthesis Evaluation — State-of-the-Art Assessment and Suggestion for a Novel Research Program

“…This insight is mainstream for related domains such as the evaluation of dialogue systems, where perceived system quality cannot be meaningfully assessed in a decontextualized fashion [3,4]. First evidence supporting this claim also for the domain of TTS evaluation has been produced by [5], who show that the same TTS material is rated differently in a crowdsourced, noninteractive MOS rating, and an MOS rating following an interaction between a human and a virtual agent in a collaborative task. Despite these insights, a meta analysis [6] revealed that the vast majority of TTS evaluations remain to rely on decontextualized listening tests, where participants score the quality of isolated sentences rather than embedding them within realistic applications or meaningful interactions.…”

“…In a similar vein, recent times have seen an increasing number of papers criticizing traditional approaches to TTS evaluation [7,5], or pointing out frequent methodological flaws such as the low validity of most TTS evaluations due to small participant numbers and a lack of diversity in the tested listener groups, especially in the light of vast individual differences between listeners [8,9], which shows stronger for some traits (age, human-likeness) than others (gender, accent origin) [10]. Generally, these investigations point out the necessity for a better conceptual framing of the perception tasks, together with larger test populations and more careful statistical approaches.…”

Proceedings of DiSS 2019

“…One practical application of the results obtained here is the extension of the hesitation insertion model for speech synthesis, which has been prototypically tested in Betz et al (2018), which did not yet take into account the structural interplay of silences and fillers. Furthermore, the hesitation model by Betz et al (2018) is centered on lengthening, which provides an elegant entry point for a synthetic hesitation interval, and reflects human speech production by making use of the pre-planned, but not-yet-uttered words in the articulatory buffer .…”

“…One practical application of the results obtained here is the extension of the hesitation insertion model for speech synthesis, which has been prototypically tested in Betz et al (2018), which did not yet take into account the structural interplay of silences and fillers. Furthermore, the hesitation model by Betz et al (2018) is centered on lengthening, which provides an elegant entry point for a synthetic hesitation interval, and reflects human speech production by making use of the pre-planned, but not-yet-uttered words in the articulatory buffer . This approach receives support by the confirmed notion of longer silences after um-type fillers: the presence of a nasal sound makes this type of filler a better candidate to smoothly initiate a hesitation interval by lengthening compared to the uh-type fillers (for hesitation lengthening distribution over phone types, cf.…”