Effects of Dimensional Input on Paralinguistic Information Perceived from Synthesized Dialogue Speech with Neural Network

Yokoyama, Masaki; Nagata, T.; Mori, Hiroki

doi:10.21437/interspeech.2018-2042

Cited by 3 publications

(1 citation statement)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, Ben-David and Shechtman proposed a prosody-controllable speech synthesis that can express expressive styles given conversational context [42]. Besides these, no work on spontaneous conversational speech synthesis was found in Interspeech conferences or SSW (Speech Synthesis Workshops) presentations, except for the authors' works [43], [44]. One reason for this might be that it has been unclear what a spontaneously speaking machine could be useful for.…”

Section: Introductionmentioning

confidence: 99%

How Does a Spontaneously Speaking Conversational Agent Affect User Behavior?

Iizuka

Mori

2022

IEEE Access

View full text Add to dashboard Cite

People treat conversational agents as mere tools and not as human-like social actors. While there has been much research on human-like agents, few studies have approached the realization of a conversational agent as a social actor from the viewpoint of speech synthesis. This study investigated the effect of synthetic voice of conversational agent trained with spontaneous speech on human interactants. Specifically, we hypothesized that humans will exhibit more social responses when interacting with conversational agent that has a synthetic voice built on spontaneous speech. Typically, speech synthesizers are built on a speech corpus where voice professionals read a set of written sentences. The synthesized speech is clear as if a newscaster were reading a news or a voice actor were playing an anime character. However, this is quite different from spontaneous speech we speak in everyday conversation. Recent advances in speech synthesis enabled us to build a speech synthesizer on a spontaneous speech corpus, and to obtain a near conversational synthesized speech with reasonable quality. By making use of these technologies, we examined whether humans produce more social responses to a spontaneously speaking conversational agent. We conducted a large-scale conversation experiment with a conversational agent whose utterances were synthesized with the model trained either with spontaneous speech or read speech. The result showed that the subjects who interacted with the agent whose utterances were synthesized from spontaneous speech tended to show shorter response time and a larger number of backchannels. The result of a questionnaire showed that subjects who interacted with the agent whose utterances were synthesized from spontaneous speech tended to rate their conversation with the agent as closer to a human conversation. These results suggest that speech synthesis built on spontaneous speech is essential to realize a conversational agent as a social actor.INDEX TERMS Conversational agents, Spontaneous speech synthesis, Human-computer interaction.

show abstract

Section: Introductionmentioning

confidence: 99%

How Does a Spontaneously Speaking Conversational Agent Affect User Behavior?

Iizuka

Mori

2022

IEEE Access

View full text Add to dashboard Cite

show abstract

End-to-end conversational speech synthesis with controllable emotions in the dimensions of pleasantness and arousal

Mori,

Nishino

2024

Acoust. Sci. & Tech.

View full text Add to dashboard Cite

We propose an end-to-end conversational speech synthesis system that allows for flexible control of emotional states defined over emotion dimensions. We extend the Tacotron 2 and VITS architectures to accept emotion dimensions as input. Initially, the model is pre-trained using a large-scale spontaneous speech corpus, followed by fine-tuning using a natural dialogue speech corpus with manually annotated perceived emotion in the form of pleasantness and arousal. Since the pretraining lacks emotion information, we explore two pre-training strategies and demonstrate that applying an emotion dimension estimator before the pre-training enhances emotion controllability. Evaluation of the synthesized speech using VITS yields a mean opinion score of 4 or higher for naturalness. Furthermore, there is a correlation of R = 0.53 for pleasantness and R = 0.89 for arousal between the given and perceived emotional states. These results underscore the effectiveness of our proposed conversational speech synthesis system with emotion control.

show abstract

Neural Conversational Speech Synthesis with Flexible Control of Emotion Dimensions

Mori

Nishino

2022

2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)

View full text Add to dashboard Cite

Effects of Dimensional Input on Paralinguistic Information Perceived from Synthesized Dialogue Speech with Neural Network

Cited by 3 publications

References 9 publications

How Does a Spontaneously Speaking Conversational Agent Affect User Behavior?

How Does a Spontaneously Speaking Conversational Agent Affect User Behavior?

End-to-end conversational speech synthesis with controllable emotions in the dimensions of pleasantness and arousal

Neural Conversational Speech Synthesis with Flexible Control of Emotion Dimensions

Contact Info

Product

Resources

About