Prediction of Who Will Be Next Speaker and When Using Mouth-Opening Pattern in Multi-Party Conversation

Abstract

We investigated the mouth-opening transition pattern (MOTP), which represents the change of mouth-opening degree during the end of an utterance, and used it to predict the next speaker and utterance interval between the start time of the next speaker’s utterance and the end time of the current speaker’s utterance in a multi-party conversation. We first collected verbal and nonverbal data that include speech and the degree of mouth opening (closed, narrow-open, wide-open) of participants that were manually annotated in four-person conversation. A key finding of the MOTP analysis is that the current speaker often keeps her mouth narrow-open during turn-keeping and starts to close it after opening it narrowly or continues to open it widely during turn-changing. The next speaker often starts to open her mouth narrowly after closing it during turn-changing. Moreover, when the current speaker starts to close her mouth after opening it narrowly in turn-keeping, the utterance interval tends to be short. In contrast, when the current speaker and the listeners open their mouths narrowly after opening them narrowly and then widely, the utterance interval tends to be long. On the basis of these results, we implemented prediction models of the next-speaker and utterance interval using MOTPs. As a multimodal-feature fusion, we also implemented models using eye-gaze behavior, which is one of the most useful items of information for prediction of next-speaker and utterance interval according to our previous study, in addition to MOTPs. The evaluation result of the models suggests that the MOTPs of the current speaker and listeners are effective for predicting the next speaker and utterance interval in multi-party conversation. Our multimodal-feature fusion model using MOTPs and eye-gaze behavior is more useful for predicting the next speaker and utterance interval than using only one or the other.

Didn't see that coming: a survey on non-verbal social human behavior forecasting

“…They proposed a light SVM method that could be deployed to any agent equipped with a camera or a depth sensor. Similarly, in multi-party conversations, Ishii et al (2019) discovered that the speaker's and listeners' mouth-opening transition patterns could be used to predict the next speaker and the time interval between the current utterance ends and the next utterance begins. To prove it, they developed a three-step system.…”

Trimodal prediction of speaking and listening willingness to help improve turn-changing modeling

“…Others have used visual features, such as overall physical motion (Chen and Harper, 2009 ; de Kok and Heylen, 2009 ; Dielmann et al, 2010 ; Roddy et al, 2018 ) near the end of a speaker's utterances or during multiple utterances. Moreover, some research has focused on detailed non-verbal behaviors, such as eye-gaze behavior (Chen and Harper, 2009 ; de Kok and Heylen, 2009 ; Huang et al, 2011 ; Jokinen et al, 2013 ; Ishii et al, 2015a , 2016a ), head movement (Huang et al, 2011 ; Ishii et al, 2015b , 2017 ), mouth movement (Ishii et al, 2019 ), and respiration (Ishii et al, 2015a , 2016b ). Specifically, information on the length of time and patterns of a speaker's gaze direction toward a listener during speaking, the amount of head movement, the patterns of the mouth opening and closing, and the amount of inspiratory volume are used as features for prediction.…”

“…However, many studies on turn-changing prediction use mainly features as mentioned above extracted from only speakers (Chen and Harper, 2009 ; de Kok and Heylen, 2009 ; Dielmann et al, 2010 ; Huang et al, 2011 ; Jokinen et al, 2013 ; Lala et al, 2018 ; Masumura et al, 2018 , 2019 ; Roddy et al, 2018 ). Several studies have used limited features and modalities of listeners, such as linguistic, eye-gaze behavior, head movement, mouse movement, and respiration as mentioned above (Ishii et al, 2015a , b , 2016a , b , 2017 , 2019 ; Masumura et al, 2018 ).…”

“…The field of human-computer interaction has long been dedicated to computational modeling of turn-changing. Many studies have focused on developing actual turn-changing (i.e., next speaker or end-of-turn) models that can predict whether turn-keeping or turn-changing will happen using participants' verbal and non-verbal behaviors (Ferrer et al, 2002 ; Schlangen, 2006 ; Chen and Harper, 2009 ; de Kok and Heylen, 2009 ; Laskowski et al, 2011 ; Kawahara et al, 2012 ; Jokinen et al, 2013 ; Holler and Kendrick, 2015 ; Ishii et al, 2015a , b , 2016a , b , 2017 , 2019 ; Lammertink et al, 2015 ; Levinson, 2016 ; Hömke et al, 2017 ; Hara et al, 2018 ; Holler et al, 2018 ; Lala et al, 2018 ; Masumura et al, 2018 , 2019 ; Roddy et al, 2018 ; Ward et al, 2018 ). These studies predicted turn-changing on the basis of verbal and non-verbal behaviors.…”

Can Prediction of Turn-management Willingness Improve Turn-changing Modeling?

“…With such knowledge, many studies have developed models for predicting actual turn-changing, i.e., whether turn-changing or turn-keeping will take place, on the basis of acoustic features [3, 6, 10, 12, 18, 26, 34, 36ś38, 43, 47, 50], linguistic features [34,37,38,43], and visual features, such as overall physical motion [3,6,8,43] near the end of a speaker's utterances or during multiple utterances. Moreover, some research has focused on detailed non-verbal behaviors such as eye-gaze behavior [3,6,18,20,24,26], head movement [18,21,22], mouth movement [23], and respiration [20,25]. However, many turn-changing prediction studies use mainly features extracted from speakers.…”