Learning multimodal behavioral models for face-to-face social interaction

Abstract

International audienceThe aim of this paper is to model multimodal perception-action loops of human behavior in face-to-face interactions. The long-term goal of this research is to give artificial agents social skills to engage believable interactions with human interlocutors. To this end, we propose trainable behavioral models that generate optimal actions given others’ perceived actions and joint goals. We first compare sequential models - in particular Discrete Hidden Markov Models (DHMMs) - with standard classifiers (SVMs and Decision Trees). We propose a modification of the initialization of the DHMMs in order to better capture the recurrent structure of the sensory-motor states. We show that the explicit state duration modeling by Hidden Semi Markov Models (HSMMs) improves prediction performance. We applied these models to parallel speech and gaze data collected from interacting dyads. The challenge was to predict the gaze of one subject given the gaze of the interlocutor and the voice activity of both. For both HMMs and HSMMs the Short-Time Viterbi concept is used for incremental decoding and generation. For the proposed models we evaluated objectively many properties in order to go beyond pure classification performance. Results show that while Incremental Discrete HMMs (IDHMMs) were more efficient than classic classifiers, the Incremental Discrete HSMMs (IDHSMMs) give best performance. This result emphasizes the relevance of state duration modeling

Learning off-line vs. on-line models of interactive multimodal behaviors with recurrent neural networks

“…The LSTM and Bi-LSTM can automatically learn contextual variables from the interaction scenario. In order to compare the efficiency of the methods, actions (FX and GT) generated offline by BiLSTM are first compared with HMM [5] and DBN [6]. In addition, online predictions of the actions by LSTM are also compared with short-term Viterbi decoding of HMM and online filter prediction of DBN.…”

“…A multimodal interactive model based on HMM was proposed in [5]. In this model, each interactive unit (IU) is modeled by one Discrete Hidden Markov Model (DHMM) that models joint multimodal sensorimotor behaviors via its hidden states.…”

“…The DHMM model with fully-connected states is implemented with 5 hidden units for each single DHMM using PMTK3 toolkit of Matlab [22]. Mihoub [5] showed that the results were not improved by using 6 or 7 unit states.…”

“…while using the conversational regime as a latent variable. Mihoub et al [5] estimated interaction units using Hidden Markov Models (HMM) to generate the gaze of an interlocutor from his own speech activity together with the gaze and speech activity of his partner. Mihoub et al [6] further showed that DBN outperform both full-and semi-HMM in predicting co-verbal behaviors in a "put that there" game.…”

Social Intelligence Modeling using Wearable Devices

“…Notice that one single CS could also sequence many sensory-motor states together. Similar concepts have been proposed in the literature of multimodal behavior modeling [23,32,35]. In our application, we used data mining techniques to explore the cognitive states of the presenter.…”