Conversational and Social Laughter Synthesis with WaveNet

Mori, Hiroki; Nagata, T.; Arimoto, Yoshiko

doi:10.21437/interspeech.2019-2131

Cited by 7 publications

(5 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order for conversational agents to be able to produce fillers or laughs, at least two research questions should be considered: how to generate them, and when to generate them. Regarding how, we are working on a research project to synthesize laughter from corpora of natural conversation [63], [64]. There are also attempts to synthesize fillers [37], [38], [65].…”

Section: Discussionmentioning

confidence: 99%

How Does a Spontaneously Speaking Conversational Agent Affect User Behavior?

Iizuka

Mori

2022

IEEE Access

View full text Add to dashboard Cite

People treat conversational agents as mere tools and not as human-like social actors. While there has been much research on human-like agents, few studies have approached the realization of a conversational agent as a social actor from the viewpoint of speech synthesis. This study investigated the effect of synthetic voice of conversational agent trained with spontaneous speech on human interactants. Specifically, we hypothesized that humans will exhibit more social responses when interacting with conversational agent that has a synthetic voice built on spontaneous speech. Typically, speech synthesizers are built on a speech corpus where voice professionals read a set of written sentences. The synthesized speech is clear as if a newscaster were reading a news or a voice actor were playing an anime character. However, this is quite different from spontaneous speech we speak in everyday conversation. Recent advances in speech synthesis enabled us to build a speech synthesizer on a spontaneous speech corpus, and to obtain a near conversational synthesized speech with reasonable quality. By making use of these technologies, we examined whether humans produce more social responses to a spontaneously speaking conversational agent. We conducted a large-scale conversation experiment with a conversational agent whose utterances were synthesized with the model trained either with spontaneous speech or read speech. The result showed that the subjects who interacted with the agent whose utterances were synthesized from spontaneous speech tended to show shorter response time and a larger number of backchannels. The result of a questionnaire showed that subjects who interacted with the agent whose utterances were synthesized from spontaneous speech tended to rate their conversation with the agent as closer to a human conversation. These results suggest that speech synthesis built on spontaneous speech is essential to realize a conversational agent as a social actor.INDEX TERMS Conversational agents, Spontaneous speech synthesis, Human-computer interaction.

show abstract

Section: Discussionmentioning

confidence: 99%

How Does a Spontaneously Speaking Conversational Agent Affect User Behavior?

Iizuka

Mori

2022

IEEE Access

View full text Add to dashboard Cite

show abstract

“…To analyze naturally occurred speech-laugh, online gaming voice chat corpus with emotional label (OGVC) [5] was used as the speech material. The dialog speech of seven out of thirteen speakers was used for this analysis because these dialogs were already annotated with laugh label and used for laughter synthesis [4]. The duration of the dialog was about 60 minutes long for each speaker and a total of seven hours of dialog were used for this analysis.…”

Section: Speech-laugh Materials 21 Corpusmentioning

confidence: 99%

Phonetic analysis on speech-laugh occurrence in spontaneous gaming dialog

Arimoto

2023

Acoust. Sci. & Tech.

Self Cite

View full text Add to dashboard Cite

“…Laughter generation in robots and agents has also been addressed but primarily in terms of animation or movement to produce realistic laughs ( Niewiadomski and Pelachaud, 2012 ; Ishi et al, 2016a ; Ishi et al, 2019 ). For the majority of agents, the range of laughter utterances is restricted by the text-to-speech system and is unable to generate speech laughs, although recent research has addressed this issue by producing a large variety of natural-sounding laughs ( Mori et al, 2019 ; Tits et al, 2020 ; Luong and Yamagishi, 2021 ).…”

Section: Related Workmentioning

confidence: 99%

Can a robot laugh with you?: Shared laughter generation for empathetic spoken dialogue

2022

View full text Add to dashboard Cite

Spoken dialogue systems must be able to express empathy to achieve natural interaction with human users. However, laughter generation requires a high level of dialogue understanding. Thus, implementing laughter in existing systems, such as in conversational robots, has been challenging. As a first step toward solving this problem, rather than generating laughter from user dialogue, we focus on “shared laughter,” where a user laughs using either solo or speech laughs (initial laugh), and the system laughs in turn (response laugh). The proposed system consists of three models: 1) initial laugh detection, 2) shared laughter prediction, and 3) laugh type selection. We trained each model using a human-robot speed dating dialogue corpus. For the first model, a recurrent neural network was applied, and the detection performance achieved an F1 score of 82.6%. The second model used the acoustic and prosodic features of the initial laugh and achieved a prediction accuracy above that of the random prediction. The third model selects the type of system’s response laugh as social or mirthful laugh based on the same features of the initial laugh. We then implemented the full shared laughter generation system in an attentive listening dialogue system and conducted a dialogue listening experiment. The proposed system improved the impression of the dialogue system such as empathy perception compared to a naive baseline without laughter and a reactive system that always responded with only social laughs. We propose that our system can be used for situated robot interaction and also emphasize the need for integrating proper empathetic laughs into conversational robots and agents.

show abstract

Conversational and Social Laughter Synthesis with WaveNet

Cited by 7 publications

References 22 publications

How Does a Spontaneously Speaking Conversational Agent Affect User Behavior?

How Does a Spontaneously Speaking Conversational Agent Affect User Behavior?

Phonetic analysis on speech-laugh occurrence in spontaneous gaming dialog

Can a robot laugh with you?: Shared laughter generation for empathetic spoken dialogue

Contact Info

Product

Resources

About