Using Respiration to Predict Who Will Speak Next and When in Multiparty Meetings

Abstract

Techniques that use nonverbal behaviors to predict turn-changing situations—such as, in multiparty meetings, who the next speaker will be and when the next utterance will occur—have been receiving a lot of attention in recent research. To build a model for predicting these behaviors we conducted a research study to determine whether respiration could be effectively used as a basis for the prediction. Results of analyses of utterance and respiration data collected from participants in multiparty meetings reveal that the speaker takes a breath more quickly and deeply after the end of an utterance in turn-keeping than in turn-changing. They also indicate that the listener who will be the next speaker takes a bigger breath more quickly and deeply in turn-changing than the other listeners. On the basis of these results, we constructed and evaluated models for predicting the next speaker and the time of the next utterance in multiparty meetings. The results of the evaluation suggest that the characteristics of the speaker's inhalation right after an utterance unit—the points in time at which the inhalation starts and ends after the end of the utterance unit and the amplitude, slope, and duration of the inhalation phase—are effective for predicting the next speaker in multiparty meetings. They further suggest that the characteristics of listeners' inhalation—the points in time at which the inhalation starts and ends after the end of the utterance unit and the minimum and maximum inspiration, amplitude, and slope of the inhalation phase—are effective for predicting the next speaker. The start time and end time of the next speaker's inhalation are also useful for predicting the time of the next utterance in turn-changing.

Improving Prediction of Speech Activity Using Multi-Participant Respiratory State

“…Furthermore, we had predicted speech activity for each respiratory cycle, whereas in the current work predictions are made every 100 ms, from the beginning of the conversation to its end. In this second respect, the current article also extends the work in [2], where predictions were made only at turn landmark locations. In addition, unlike the authors of [2], who compared their system to a random baseline, the current article compares the contribution of breathing in the context of multi-participant speech activity to a baseline trained on multi-participant speech activity alone.…”

“…There were also some indications that turn-initial inhalation was deeper than inhalations made later in the turn [4,5]; however, these findings were not confirmed by other studies [2], or were only observed in scripted dialogues [4]. By contrast, [2] found that inhalation tends to be deeper in turn-holding than in turn-changing pauses, although the size of this effect seemed to be rather small. By including an additional category of backchannel-like utterances in our own work we have been able to identify consistent variation in inhalation amplitude across turn-categories [3].…”

“…In addition, exhalations were shown to be longer before speaker changes than before turn continuations [4], and preparing to take the turn was associated with an increased inhalation amplitude compared to the silent breathing pattern [2]. There were also some indications that turn-initial inhalation was deeper than inhalations made later in the turn [4,5]; however, these findings were not confirmed by other studies [2], or were only observed in scripted dialogues [4]. By contrast, [2] found that inhalation tends to be deeper in turn-holding than in turn-changing pauses, although the size of this effect seemed to be rather small.…”

“…In particular, several studies [1,2,3] have identified temporal compression, whereby the lag preceding and following the inhalation as well as the duration of the inhalation itself are reduced, a practice interpreted as a strategy to minimize the likelihood of pause interruption. In addition, exhalations were shown to be longer before speaker changes than before turn continuations [4], and preparing to take the turn was associated with an increased inhalation amplitude compared to the silent breathing pattern [2]. There were also some indications that turn-initial inhalation was deeper than inhalations made later in the turn [4,5]; however, these findings were not confirmed by other studies [2], or were only observed in scripted dialogues [4].…”

“…The only exception is Ishii et al [2], who tested a three-stage prediction model. At each utterance offset, the model predicts: (1) whether the same speaker (versus another) is going to continue, (2) in case of speaker change, which of the three conversation partners is going to take the turn, and (3) when the next utterance is going to start (whether from the previous speaker or one of the listeners).…”