Evaluation of Real-time Deep Learning Turn-taking Models for Multiple Dialogue Scenarios

Abstract

The task of identifying when to take a conversational turn is an important function of spoken dialogue systems. The turn-taking system should also ideally be able to handle many types of dialogue, from structured conversation to spontaneous and unstructured discourse. Our goal is to determine how much a generalized model trained on many types of dialogue scenarios would improve on a model trained only for a specific scenario. To achieve this goal we created a large corpus of Wizard-of-Oz conversation data which consisted of several different types of dialogue sessions, and then compared a generalized model with scenario-specific models. For our evaluation we go further than simply reporting conventional metrics, which we show are not informative enough to evaluate turn-taking in a real-time system. Instead, we process results using a performance curve of latency and false cut-in rate, and further improve our model's real-time performance using a finite-state turn-taking machine. Our results show that the generalized model greatly outperformed the individual model for attentive listening scenarios but was worse in job interview scenarios. This implies that a model based on a large corpus is better suited to conversation which is more user-initiated and unstructured. We also propose that our method of evaluation leads to more informative performance metrics in a real-time system. CCS CONCEPTS • Human-centered computing → Human computer interaction (HCI); • Computing methodologies → Supervised learning by classification; Discourse, dialogue and pragmatics; • General and reference → Evaluation;

A Neural Turn-Taking Model without RNN

“…We experimentally evaluated the proposed model on a Japanese conversation corpus (identical as that previously used [10]) that consists of four types of conversations and over 30, 000 utterances. The type and the number of the sessions are shown in Table 1.…”

“…Such hard-coded models are difficult to transfer to other languages/cultures since they are culture-dependent. Data-driven methods such as finite state machine-based [7] and neural network-based models [8,9,10] have also been proposed in recent years. These works use feature sequences extracted from both text and speech signals.…”

“…In the proposed model, the lexical information is processed by a capsule network [19] with a convolutional layer, where the acoustic information is handled by a dilated convolutional network with ResNets [17] as its building blocks. We experimentally compared our proposed model with two RNN-based models (nested [8] and stacked [10]) on a Japanese conversational corpus that included four types of conversations: dating, job interviews, attentive listening, and at reception-counter conversations. Our experimental results show that the proposed non-RNN model outperformed RNN-based networks in a turntaking estimation task.…”

Turn-Taking Prediction Based on Detection of Transition Relevance Place

“…This model detects TRP at the end of IPU, based on prosodic and linguistic information of the preceding utterance. We used a hierarchical model of LSTM where each kind of feature is modeled by an individual LSTM and the outputs of those LSTMs are concatenated and fed into to a linear layer that outputs the posterior probability of the output label [29], as shown in Figure 2. The reference labels are binary corresponding to the TRP labels annotated in Section 3.…”

“…The prediction model was based on conditional random field [16], support vector machines [24], and neural networks [25]. A recent approach is to use recurrent neural networks such as long shortterm memory (LSTM), which can handle long-range context of the input sequence, and it achieved higher accuracy than conventional methods [15,26,19,27,28,29,30]. However, the performance is still low in natural conversations.…”