Towards an Unsupervised Entrainment Distance in Conversational Speech Using Deep Neural Networks

Abstract

Entrainment is a known adaptation mechanism that causes interaction participants to adapt or synchronize their acoustic characteristics. Understanding how interlocutors tend to adapt to each other's speaking style through entrainment involves measuring a range of acoustic features and comparing those via multiple signal comparison methods. In this work, we present a turn-level distance measure obtained in an unsupervised manner using a Deep Neural Network (DNN) model, which we call Neural Entrainment Distance (NED). This metric establishes a framework that learns an embedding from the population-wide entrainment in an unlabeled training corpus. We use the framework for a set of acoustic features and validate the measure experimentally by showing its efficacy in distinguishing real conversations from fake ones created by randomly shuffling speaker turns. Moreover, we show real world evidence of the validity of the proposed measure. We find that high value of NED is associated with high ratings of emotional bond in suicide assessment interviews, which is consistent with prior studies.

Linking emotions to behaviors through deep transfer learning

“…Behavior quantification from speech Behavioral signal processing (BSP) Georgiou et al, 2011b) can play a central role in informing human assessment and decision making, especially in assisting domain specialists to observe, evaluate and identify domain-specific human behaviors exhibited over longer time scales. For example, in couples therapy (Black et al, 2013;Nasir et al, 2017b), depression (Gupta et al, 2014;Nasir et al, 2016;Stasak et al, 2016;Tanaka et al, 2017) and suicide risk assessment (Cummins et al, 2015;Venek et al, 2017;Nasir et al, 2018Nasir et al, , 2017a, behavior analysis systems help psychologists observe and evaluate domain-specific behaviors during interactions. Li et al (2016) proposed sparsely connected and disjointly trained deep neural networks to deal with the low-resource data issue in behavior understanding.…”

“…Moreover, many other aspects of behavior, such as entrainment, turn-taking duration, pauses, non-verbal vocalizations, and influence between interlocutors, can be incorporated. Many such additional features can be similarly developed on different data and employed as primitives; for example entrainment measures can be trained through unlabeled data (Nasir et al, 2018). Furthermore, we expect that the results of behavior classification accuracy maybe be further improved through improved architectures, parameter tuning, and data engineering for each behavior of interest.…”

Predicting couple therapy outcomes based on speech acoustic features

“…More recently, dynamic models to characterize the changes in behavior of couples during interactions have been proposed–both in acoustic [ 67 ] and lexical modalities [ 12 ], and extensions of the lexical work to produce more robust methods have been introduces within a neural-net framework [ 68 ]. Finally, some early results from our current work on prediction of marital outcome from acoustic features were presented in [ 69 ] with a simpler methodology and basic analyses. In the current work, we developed a improved framework that extracts both short-term and long-term temporal changes in acoustic features.…”

Modeling Interpersonal Linguistic Coordination in Conversations Using Word Mover’s Distance

“…When people engage in conversations in social settings, they tend to coordinate with each other and show similar behavior in various modalities. This tendency, known as entrainment or coordination, is exhibited through facial expressions [1], head-motion [2], vocal patterns (vocal entrainment) [3,4], as well as the use of language (linguistic coordination) [5]. Linguistic coordination is a well-established phenomenon in both spoken and written communication that has many collaborative benefits.…”