A Neural Turn-Taking Model without RNN

Abstract

Sequential data such as speech and dialogs are usually modeled by Recurrent Neural Networks (RNN) and derivatives since the information can travel through time with such architecture. However, disadvantages exist with the use of RNNs, including the limited depth of neural networks and the GPU's unfriendly training process. Estimating the timing of turn-taking is a critical feature of dialog systems. Such tasks require knowledge about past dialog contexts and have been modeled using RNNs in several studies. In this paper, we propose a non-RNN model for the timing estimation of turn-taking in dialogs. The proposed model takes lexical and acoustic features as its input to predict a turn's end. We conducted experiments on four types of Japanese conversation datasets and show that with proper neural network designs, the long-term information in a dialog could propagate without a recurrent structure. The proposed model outperformed canonical RNN-based architectures on a turn-taking estimation task.

Gated Multimodal Fusion with Contrastive Learning for Turn-taking Prediction in Human-robot Dialogue

“…However, this approach is limited in its naturalness that the fixed threshold can potentially be too short (frequent interruptions) or too long (awkward pauses). To address this problem, machine learning methods have gained popularity since 1970s [3, 4, inter alia], and models based on inter-pausal unit (IPU), an audio segment followed by silence longer than 200 milliseconds, have mostly been studied recently because of its simplicity in practice [5,6]. For a specific IPU, various cues across modalities, such as prosody, semantics, syntax, breathing, gesture, and eye-gaze can be extracted and integrated to determine whether this turn is yielded or not [7,8].…”